dmkitui / arachneserver / 60 / 2

Ran 12 Nov 2018 07:42AM UTC

Files 7

Run time 0s

Badge

Embed ▾

Committed 12 Nov 2018 07:41AM UTC coverage: 84.034% (-10.1%) from 94.149%

Job # TOXENV=scrapy-1.5.0

Build Type

push

travis-ci

Committed by

web-flow

Commit Message

Ch spider status (#5)

* Update user agent

* Add checking if a spider is already running, and return appropriate message, or start the crawler if it is not runnning

* Add closed_spider callback

* Add empty dictionary to hold running spider state

* Update USER_AGENT information

* Refactor the run_spider_endpoint while checking for the status of a spider, to use the now global SPIDER_STATUS dict

* Add `ApplicationData` extension to default EXTENSIONS. This extension is used for reading and loading data about available spiders, and writing spider statistics to a persistent.

* Add the ApplicationData extension to handle reading and writing spider statistics to and from file

* Add the spider_info method for handling the reading and writing spider statistics to and from file

* Refactor to move statiscs gathering to an extension.

* Add csv file for spider data persistence

Run Details

200 of 238 relevant lines covered (84.03%)

0.84 hits per line

dmkitui / arachneserver / 60 / 2

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 60.2 (TOXENV=scrapy-1.5.0)

dmkitui / arachneserver / 60 / 2

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 60.2 (TOXENV=scrapy-1.5.0)

README BADGES
x