berkmancenter / mediacloud / 4523
59%
master: 70%

LAST BUILD BRANCH: release
DEFAULT BRANCH: master
Ran
Jobs 4
Files 212
Run time 14min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
4523

push

travis-ci

pypt 
Test normalized URL in is_http_url()

is_http_url() is a helper that we use for making sure that a string
looks like a valid URL before attempting to fetch it.

One could say that this is inexact science as the only way to find out
whether we can fetch the URL is to try to actually fetch it and catch
exceptions and whatnot. However, due to the quite big feature set of the UserAgent
class (max. size, max. redirects, HTTP / HTML redirects, backoff
retries, ...), it would be risky to add a single try-catch block around
every call to "requests" because that could hide potential programming
errors that we've made in this very young class by porting it to Python.

So, callers to UserAgent class are expected to call is_http_url() first
for their string arguments to make sure that the parameter is an URL
(otherwise one of the UserAgent's methods will do that for them and
raise an exception).

Before, is_http_url() was using furl as one of the methods of testing
the URL; however, it wasn't verifying all of its accessors, i.e. it
wasn't testing whether an URL can be constructed back from a furl
object. Because of that, is_http_url() was reporting the following URL
as valid:

http://michigan-state-football-sexual-assault-charges-arrest-players-names

...but then "requests" would normalize the URL by adding a slash to it
and subsequent calls to is_http_url() would fail for URL:

http://michigan-state-football-sexual-assault-charges-arrest-players-names/

In this commit (PR), is_http_url() stringifies parsed furl object back
to an URL and also tries to validate a normalized version of the URL.

(Hopefully) fixes #224.

8601 of 14621 relevant lines covered (58.83%)

1908.57 hits per line

Jobs
ID Job ID Ran Files Coverage
1 4523.1 (MC_TEST_CHUNK=1) 0
40.06
 Travis Job 4523.1
2 4523.2 (MC_TEST_CHUNK=2) 0
40.86
 Travis Job 4523.2
3 4523.3 (MC_TEST_CHUNK=3) 0
32.68
 Travis Job 4523.3
4 4523.4 (MC_TEST_CHUNK=4) 0
40.49
 Travis Job 4523.4
Source Files on build 4523
Detailed source file information is not available for this build.