berkmancenter / mediacloud / 5641

70%

push

travis-ci

optimize query that provides downloads to the crawler

The existing query that provides pending downlopads to the crawler
was behaving poorly for the case that we have lots of pending downloads
and that they are over concentrated in a single source.  The result
was that the vast majority of the pending downloads getting into the
intermediate queue were from a couple dozen sites.  The downloads
from that intermediate queue were being throttled to avoid downloading
more than once per second from a given site, so the crawler was getting
very few downloads into the the fetchers.

The updated query limits the number of downloads returned to one
million but sorts those downloads by the rank for the given site.  This
approach greatly reduces the impact of the couple dozen sites with tens
of thousands of downloads in the queue, allowing the crawler to work
through the download queue much more quickly.

5 of 5 new or added lines in 1 file covered. (100.0%)

13989 of 22437 relevant lines covered (62.35%)

634.18 hits per line