1
57%
master: 70%

Ran 17 Apr 2017 08:26PM UTC

Files 183

Run time 15s

Badge

Embed ▾

Committed 17 Apr 2017 07:39PM UTC coverage: 56.525%. First build

Job # 3570.1

Build Type

push

travis-ci

Committed by

hroberts

Commit Message

requeue media locked extractor jobs to avoid media congestion

Our extractor job pool has the potential to get stuck on a single choke point in
the sentence deduping code, where it has to wait for a lock on the media_id
story.  If the extractor job pool gets a long series of stories with the same media_id,
we can end up waiting for the pool to process one of those jobs at a time until the
media_id series is cleared.

This commit mitigates this issue by checking at the very beginning of the extract_and_vector
job whether the media_id of the story is currently locked, using a non-blocking call.  If the
media_id is locked, the job just requeues itself under low priority and exits.  This will still
allow for a brief backlog when a few stories from the same media_id are initially put into the
queue, but it should clear up any of those backlogs within a couple of minutes, instead of
potentially blocking for a couple of days, as has happened a couple of times.

Run Details

8394 of 14850 relevant lines covered (56.53%)

1581.89 hits per line

berkmancenter / mediacloud / 3570 / 1
57%
master: 70%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 3570.1

berkmancenter / mediacloud / 3570 / 1 57% master: 70%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 3570.1

berkmancenter / mediacloud / 3570 / 1
57%
master: 70%

README BADGES
x