berkmancenter / mediacloud / 2054

48%

master: 70%

push

travis-ci

Don't extract / CoreNLP-annotate / Bit.ly-process old reextracted stories

Current variant of the reextractor script ("reextract_stories_without_readability_tag.pl") added stories to extractor's queue with Bit.ly processing and CoreNLP annotation enabled, like for any other story to be extracted. This behavior filled up CoreNLP and Bit.ly queues pretty quickly which is not a wanted behavior (we just want to reextract stories).

Reextractor script stopped adding stories to the extraction queue at story with stories_id = 90640349. So, as a lazy way to skip through the old job broker queue items which would have been extracted + annotated with CoreNLP + added to Bit.ly processing queue (instead of just extracted, what we want), ignore all jobs that are older than the defined "stories_id" and don't have any extra flags (e.g. "skip_bitly_processing") set.

This commit is to be reverted when the old erroneous job broker queue gets emptied out.

20 of 20 new or added lines in 3 files covered. (100.0%)

6883 of 14327 relevant lines covered (48.04%)

390.35 hits per line