berkmancenter / mediacloud / 4523 / 1
59%
master: 70%

LAST BUILD BRANCH: release
DEFAULT BRANCH: master
Ran
Files 204
Run time 32s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 40.06%. First build
MC_TEST_CHUNK=1

push

travis-ci

pypt 
Test normalized URL in is_http_url()

is_http_url() is a helper that we use for making sure that a string
looks like a valid URL before attempting to fetch it.

One could say that this is inexact science as the only way to find out
whether we can fetch the URL is to try to actually fetch it and catch
exceptions and whatnot. However, due to the quite big feature set of the UserAgent
class (max. size, max. redirects, HTTP / HTML redirects, backoff
retries, ...), it would be risky to add a single try-catch block around
every call to "requests" because that could hide potential programming
errors that we've made in this very young class by porting it to Python.

So, callers to UserAgent class are expected to call is_http_url() first
for their string arguments to make sure that the parameter is an URL
(otherwise one of the UserAgent's methods will do that for them and
raise an exception).

Before, is_http_url() was using furl as one of the methods of testing
the URL; however, it wasn't verifying all of its accessors, i.e. it
wasn't testing whether an URL can be constructed back from a furl
object. Because of that, is_http_url() was reporting the following URL
as valid:

http://michigan-state-football-sexual-assault-charges-arrest-players-names

...but then "requests" would normalize the URL by adding a slash to it
and subsequent calls to is_http_url() would fail for URL:

http://michigan-state-football-sexual-assault-charges-arrest-players-names/

In this commit (PR), is_http_url() stringifies parsed furl object back
to an URL and also tries to validate a normalized version of the URL.

(Hopefully) fixes #224.

5757 of 14371 relevant lines covered (40.06%)

457.36 hits per line

Source Files on job 4523.1 (MC_TEST_CHUNK=1)