berkmancenter / mediacloud / 6114

70%

master: 70%

Pull #556

travis-ci

Add tables, API client, Celery worker to fetch SimilarWeb stats

We call Total Traffic API
(https://www.similarweb.com/corp/developer/estimated_visits_api) for
every domain in "media" and store those stats in
"similarweb_estimated_visits" table.

By default, API calls are made with "main_domain_only" set to false,
i.e. we fetch aggregate visit count for both the queried domain and its
subdomains.

Due to the facts that:

* URL -> domain function is "greedy", i.e. sometimes chops off
meaningful subdomain;
* We have a lot of duplicates in "media";
* We have a limited number of SimilarWeb API calls;

We decouple the domain for which we fetch the stats, and media source
which has initiated the stats fetch for a domain; we link the two using
the "media_similarweb_domains_map" table. API client also avoids
refetching stats for domains for which the stats were already
(attempted at being) fetched.

Lastly, the stats fetcher Celery worker exit(1)s on errors that it deems
to be unrecoverable. We don't want to waste precious API calls, and if
one of the requests fails unrecoverably and in an unexpected manner,
it's better to just let the worker die and look into what has caused the
error.

References #550.

126 of 126 new or added lines in 6 files covered. (100.0%)

14382 of 19826 relevant lines covered (72.54%)

250.41 hits per line