1
51%
master: 51%

Ran 07 Sep 2014 11:14PM UTC

Files 13

Run time 7s

Badge

Embed ▾

Committed 07 Sep 2014 11:13PM UTC coverage: 82.553% (+6.2%) from 76.309%

Job # 44.1

Build Type

Pull #12

travis-ci

Committed by

Blahah

Commit Message

Complete overhaul of thresher architecture

This is the first, and most major step in a complete overhaul of thresher.
The purpose of this is to support the current and near-future needs of
scraperJSON, based on revisiting the design and incorporating a lot of
user feedback.

Major changes:

- all scraping functionality has been moved to the Scraper class
- the Thresher class now only handles selecting a scraper by URL, and running it
- ScraperBox class holds a collection of scrapers and can match them to URLs
- all logging has been removed and the entire module now operates using events

scraperJSON features implemented:

- elements can be nested (fixes #2 and ContentMine/scraperJSON#3)
- elements can depend on 'following' the captured URLs from other elements (fixes #6)
- URLs are resolved (and all redirects followed) before scraping (fixes #10)
- headless pre-rendering is no longer default (for a massive speed/efficiency increase)

Pull Request Pull Request #12: Overhaul

Run Details

388 of 470 relevant lines covered (82.55%)

10.15 hits per line

ContentMine / thresher / 44 / 1
51%
master: 51%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 44.1

ContentMine / thresher / 44 / 1 51% master: 51%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 44.1

ContentMine / thresher / 44 / 1
51%
master: 51%

README BADGES
x