3
70%
master: 70%

Ran 11 Sep 2018 10:53PM UTC

Files 331

Run time 36s

Badge

Embed ▾

Committed 11 Sep 2018 09:58PM UTC coverage: 51.279% (-0.05%) from 51.331%

Job # MC_TEST_CHUNK=3

Build Type

push

travis-ci

Committed by

hroberts

Commit Message

optimize query that provides downloads to the crawler

The existing query that provides pending downlopads to the crawler
was behaving poorly for the case that we have lots of pending downloads
and that they are over concentrated in a single source.  The result
was that the vast majority of the pending downloads getting into the
intermediate queue were from a couple dozen sites.  The downloads
from that intermediate queue were being throttled to avoid downloading
more than once per second from a given site, so the crawler was getting
very few downloads into the the fetchers.

The updated query limits the number of downloads returned to one
million but sorts those downloads by the rank for the given site.  This
approach greatly reduces the impact of the couple dozen sites with tens
of thousands of downloads in the queue, allowing the crawler to work
through the download queue much more quickly.

Run Details

11068 of 21584 relevant lines covered (51.28%)

295.87 hits per line

berkmancenter / mediacloud / 5641 / 3
70%
master: 70%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 5641.3 (MC_TEST_CHUNK=3)

berkmancenter / mediacloud / 5641 / 3 70% master: 70%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 5641.3 (MC_TEST_CHUNK=3)

berkmancenter / mediacloud / 5641 / 3
70%
master: 70%

README BADGES
x