berkmancenter / mediacloud / 5690

63%

master: 70%

push

travis-ci

Fetch story sentences in chunks (1000 stories_id at once) for word2vec

Depending on the snapshot size, all attempts to join story_sentences to
snap.stories (with WHERE or INNER JOIN, and with or without a temporary
table) might lead to sequential scan on story_sentences.

Instead, fetch sentences in chunks (sentences for up to 1000 stories per
chunk) and feed them to word2vec.

References #348.

References #481.

27 of 27 new or added lines in 1 file covered. (100.0%)

14062 of 22456 relevant lines covered (62.62%)

632.99 hits per line