6113
70%
master: 70%

Ran 06 Mar 2019 03:27PM UTC

Jobs 2

Files 343

Run time 548min

Badge

Embed ▾

pending completion

Build # 6113

Build Type

push

travis-ci

Committed by

pypt

Commit Message

Add tables, API client, Celery worker to fetch SimilarWeb stats

We call Total Traffic API
(https://www.similarweb.com/corp/developer/estimated_visits_api) for
every domain in "media" and store those stats in
"similarweb_estimated_visits" table.

By default, API calls are made with "main_domain_only" set to false,
i.e. we fetch aggregate visit count for both the queried domain and its
subdomains.

Due to the facts that:

* URL -> domain function is "greedy", i.e. sometimes chops off
meaningful subdomain;
* We have a lot of duplicates in "media";
* We have a limited number of SimilarWeb API calls;

We decouple the domain for which we fetch the stats, and media source
which has initiated the stats fetch for a domain; we link the two using
the "media_similarweb_domains_map" table. API client also avoids
refetching stats for domains for which the stats were already
(attempted at being) fetched.

Lastly, the stats fetcher Celery worker exit(1)s on errors that it deems
to be unrecoverable. We don't want to waste precious API calls, and if
one of the requests fails unrecoverably and in an unexpected manner,
it's better to just let the worker die and look into what has caused the
error.

References #550.

Run Details

13704 of 19164 relevant lines covered (71.51%)

177.82 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
1	6113.1 (MC_TEST_CHUNK=1)	06 Mar 2019 03:32PM UTC	335	69.71	Travis Job 6113.1
2	6113.2 (MC_TEST_CHUNK=2)	06 Mar 2019 03:27PM UTC	330	60.59	Travis Job 6113.2

berkmancenter / mediacloud / 6113
70%
master: 70%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 6113

berkmancenter / mediacloud / 6113 70% master: 70%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 6113

berkmancenter / mediacloud / 6113
70%
master: 70%

README BADGES
x