TheTorProject / ooni-probe / 1638 / 4
71%
master: 71%

DEFAULT BRANCH: master
Ran
Files 64
Run time 4s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 69.32% (+0.3%) from 68.996%
TWISTED=Twisted==14.0.2

push

travis-ci

hellais 
Feature/web connectivity (#457)

* Update the tests to be consistent with the new data formats

* Maintain backward compatibility

* First stab at implementing the web_connectivity test

* Progress on implementing web_connectivity test

* Implement basic web connectivity test

* Various hacky fixes to enforce correct report format and handling of report_id

* Add detection for accessibility of the website

* Minor refactoring of the test

* Fix report uploader to work with JSON

* Add better exception handling to deckgen
* We should actually be using POST since PUT is indempotent
* Debugging of backward and forward data format compatibility
* Better integrity checks in web_consistency

* Set the default DNS discovery test helper

* Clean up how we handle the reportId and remove unused functions

* [refactoring] Change signature of NetTest() class

* [feature] Add setUpClass method to NetTestCase

* Many improvements to web_connectivity test

* Run the probe resolver detection inside of the setupClass
* Display a summary at the end of a test run summarising the results
* Make the blocking detection logic more robust

* Add check on equality of control and experiment DNS responses

* [refactoring] Change signature of Deck to be more robust

Do more sanity checks on bouncer address at instantiation time

* Add support for computing header difference and follow redirects

* Make the blocking detection heuristics more strict

* Also record all the redirects

* Add monkey patch for bug in twisted RedirectAgent: https://twistedmatrix.com/trac/ticket/8265

* Use settings.Config.ooni_home as the data directory (to avoid broken paths when using sudo or setting environment vars)

* Improvements to the output of the web_connectivity test

* Make the geoip lookup code more robust

* Implement unittest to check if tests that fail twice

This is to ensure that net tests that eventually succeed, but fail sometimes
will run properly until the end.

* Improve detection logic for web_connectivity

* Fix printing of exception log

* Fix some issues with the rebase

* Ignore empty charset values

* Improve charset detection regexp

* Add the redirected responses in inverted order

This means the first item in the list will be the final response, while the
last will be the first response.

* Remove dead code

* Patch twisted HTTPClientParser to accept HTTP response lines with a missing reason

* Take also into consideration if the HTTP headers match in detecting blockpages.

This significantly brings down the false positive ratio. Though there are still
some instances of sites being mis-reported as blocked.

Examples of these are sites that have different web servers for sites in the
various languages.

* Start adding support for HTTPS, Cloudfronted test helpers and collectors

* Add routines to verify which collectors and test helpers are reachable

* Merciless refactoring of the reporting logic

* Remove dumb logic

* Make the test-helper-alternate key work when speaking to backend

* Rename advanced option insecure_collector to insecure_backend

* Refactor isReachable() to only return booleans

* Bugfixing

* Refactor code for setting of tor port

* Clean up the output of ooniprobe

* Make OneShotFactory not be noisy

* Don't log an error when we are just connecting to a backend to check if it's
  up.

* Various improvement related to backend_client

* Include web_connectivity in default generated deck

* Fix serialisation of HTTP Headers

* Support setting collector and bouncer in oonireport upload

* Don't require the bouncer key to be present

* Change test for setupCollector() function

* Add support for Gzip Decoding in web connectivity test

* Ensure the input key is properly set

* Improve readability of the blockpage detection check

* Consider valid onion addresses also when they don't use the httpo:// schema

Based on feedback from @willscott

* Fix invalid onion address in unittests

* Address feedback by @bassosimone

* Distinguish between http-failure and http-diff

* Update description of web_connectivity test

* Various consistency and stylistic improvements

* Compare CNAME responses as well

* Expand the heuristics for detecting blockpages in HTTP response

* Extract the title of the response

* Consider only common HTTP headers

* Inject the correct url when following redirects

* Match status codes only when the control status code is not an error

* Add a timeout to the web_connectivity test to avoid the test timing out before it's done

* Share code from ooni-backend via the ooni.common module

* Address feedback by @bassosimone

* Test the next words if the word is less than 5 chars in web_connectivity
  test.

* Move the charset detection charset into the correct module and change the
  unittest accordingly.

* Include ooni.common in setup.py

* Various code style fixes

* Write a msg when we perform the control lookup

Be more lax when detecting header key similarity

3719 of 5365 relevant lines covered (69.32%)

0.69 hits per line

Source Files on job 1638.4 (TWISTED=Twisted==14.0.2)