binux / pyspider / 584

84%

push

travis-ci

use smaller connect_timeout

When having temporary network interruption, requests will hang at
connecting state, at same time, scheduler will continue dispatching new
tasks. After network recover, all request sent out in bulk, which may
cause IP ban.
A smaller connect_timeout will reduce the maximum number of requests in
connecting state when network interruption happends.

connect_timeout can be set via self.crawl, and timeout will no longer
affect connect_timeout.

2 of 2 new or added lines in 1 file covered. (100.0%)

4870 of 5446 relevant lines covered (89.42%)

3.46 hits per line