ContentMine / thresher

83%

master: 51%

LAST BUILD ON BRANCH follow-on

branch: follow-on

CHANGE BRANCH

x

Reset

• follow-on
• depfree
• fix-hyphen-urls
• master
• v0.0.1
• v0.0.2
• v0.0.3
• v0.0.5
• v0.0.6
• v0.0.7
• v0.0.9
• v0.1.10
• v0.1.11

Build # 43

Build Type

push

travis-ci

Committed by Blahah

Commit Message

Complete overhaul of thresher architecture

This is the first, and most major step in a complete overhaul of thresher.
The purpose of this is to support the current and near-future needs of
scraperJSON, based on revisiting the design and incorporating a lot of
user feedback.

Major changes:

- all scraping functionality has been moved to the Scraper class
- the Thresher class now only handles selecting a scraper by URL, and running it
- ScraperBox class holds a collection of scrapers and can match them to URLs
- all logging has been removed and the entire module now operates using events

scraperJSON features implemented:

- elements can be nested (fixes #2 and ContentMine/scraperJSON#3)
- elements can depend on 'following' the captured URLs from other elements (fixes #6)
- URLs are resolved (and all redirects followed) before scraping (fixes #10)
- headless pre-rendering is no longer default (for a massive speed/efficiency increase)

Run Details

388 of 470 relevant lines covered (82.55%)

10.15 hits per line