3570
57%
master: 70%

Ran 17 Apr 2017 08:26PM UTC

Jobs 1

Files 183

Run time 16s

Badge

Embed ▾

pending completion

Build # 3570

Build Type

push

travis-ci

Committed by

hroberts

Commit Message

requeue media locked extractor jobs to avoid media congestion

Our extractor job pool has the potential to get stuck on a single choke point in
the sentence deduping code, where it has to wait for a lock on the media_id
story.  If the extractor job pool gets a long series of stories with the same media_id,
we can end up waiting for the pool to process one of those jobs at a time until the
media_id series is cleared.

This commit mitigates this issue by checking at the very beginning of the extract_and_vector
job whether the media_id of the story is currently locked, using a non-blocking call.  If the
media_id is locked, the job just requeues itself under low priority and exits.  This will still
allow for a brief backlog when a few stories from the same media_id are initially put into the
queue, but it should clear up any of those backlogs within a couple of minutes, instead of
potentially blocking for a couple of days, as has happened a couple of times.

Run Details

8394 of 14850 relevant lines covered (56.53%)

1581.89 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
1	3570.1	17 Apr 2017 08:26PM UTC	0	56.53	Travis Job 3570.1

Source Files on build 3570

Detailed source file information is not available for this build.

berkmancenter / mediacloud / 3570
57%
master: 70%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 3570

berkmancenter / mediacloud / 3570 57% master: 70%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 3570

berkmancenter / mediacloud / 3570
57%
master: 70%

README BADGES
x