dmkitui / arachneserver / 60

push

travis-ci

Ch spider status (#5)

* Update user agent

* Add checking if a spider is already running, and return appropriate message, or start the crawler if it is not runnning

* Add closed_spider callback

* Add empty dictionary to hold running spider state

* Update USER_AGENT information

* Refactor the run_spider_endpoint while checking for the status of a spider, to use the now global SPIDER_STATUS dict

* Add `ApplicationData` extension to default EXTENSIONS. This extension is used for reading and loading data about available spiders, and writing spider statistics to a persistent.

* Add the ApplicationData extension to handle reading and writing spider statistics to and from file

* Add the spider_info method for handling the reading and writing spider statistics to and from file

* Refactor to move statiscs gathering to an extension.

* Add csv file for spider data persistence

54 of 54 new or added lines in 4 files covered. (100.0%)

193 of 238 relevant lines covered (81.09%)

1.65 hits per line