berkmancenter / mediacloud
62%
master: 70%

Repo Added 11 Jul 2014 12:47PM UTC

Build # 5754

Committed by

Commit Message

Strip non-printable control characters before extracting

A certain download (downloads_id = 1620866179):

```
https://www.facebook.com/zuck?hc_ref=ARTIabbsED4833Xhh0R0ynoiQkRaK5MjwDzGp8AaJIyir_RMwcu9K2R-TqXg3tZzkGo&fref=nf&pnref=story
```

managed to return ~11 MB of data, most of which is weird sequences of
control characters:

```
0004f040: 673e 3c61 2068 7265 663d 2268 7474 7073  g><a href="https
0004f050: 3a2f 2f69 642d 6964 2e66 6163 6562 6f6f  ://id-id.faceboo
0004f060: 6b2e 636f 6d2f 7365 7074 6979 616e 692e  k.com/septiyani.
0004f070: 6265 6c6c 6122 3e4d 6172 6b20 5a75 636b  bella">Mark Zuck
0004f080: 6572 6265 7267 3c2f 613e 3c2f 7374 726f  erberg</a></stro
0004f090: 6e67 3e3c 2f64 6976 3e3c 6469 7620 636c  ng></div><div cl
0004f0a0: 6173 733d 2270 726f 6669 6c65 4672 6965  ass="profileFrie
0004f0b0: 6e64 7354 6578 7420 7072 6f66 696c 6546  ndsText profileF
0004f0c0: 7269 656e 6473 4279 6c69 6e65 2066 736d  riendsByline fsm
0004f0d0: 2066 776e 2066 6367 223e 28e3 8080 e380   fwn fcg">(.....
0004f0e0: 80e3 8080 20e3 8080 e380 80e3 8080 20e3  .... ......... .
0004f0f0: 8080 e380 80e3 8080 20e3 8080 e380 80e3  ........ .......
0004f100: 8080 20e3 8080 e380 80e3 8080 20e3 8080  .. ......... ...
0004f110: e380 80e3 8080 20e3 8080 e380 80e3 8080  ...... .........
0004f120: 20e3 8080 e380 80e3 8080 20e3 8080 e380   ......... .....
0004f130: 80e3 8080 20e3 8080 e380 80e3 8080 20e3  .... ......... .
0004f140: 8080 e380 80e3 8080 20e3 8080 e380 80e3  ........ .......
0004f150: 8080 20e3 8080 e380 80e3 8080 20e3 8080  .. ......... ...
```

(The 0xe3 0x80 0x80 goes on for the rest of the returned file.)

Is this a UTF-8 decoding error on our end? A buffer overflow somewhere?
Dunno.

Readability is not too happy with the control character sequence and got
stuck at it. So:

* Strip ASCII control characters (except for "\n", "\r" and "\t") before
extracting;
* Strip varied whitespace sequences (for example, not only "\t\t\t", but
also "\t\n \t\n " before extracting).

9 of 9 new or added lines in 2 files covered. (100.0%)

14190 of 22752 relevant lines covered (62.37%)

312.27 hits per line

22752 RELEVANT LINES 14190 COVERED LINES

312.27 HITS PER LINE

Builds	Branch	Commit	Type	Ran	Committer	Via	Coverage
5754	extractor_strip_control_chars	Strip non-printable control characters before extracting A certain download (downloads_id = 1620866179): ``` https://www.facebook.com/zuck?hc_ref=ARTIabbsED4833Xhh0R0ynoiQkRaK5MjwDzGp8AaJIyir_RMwcu9K2R-TqXg3tZzkGo&fref=nf&pnref=story ``` manage...	Pull #490	05 Oct 2018 02:18PM UTC	web-flow	travis-ci	pending completion
5753	extractor_strip_control_chars	Strip non-printable control characters before extracting A certain download (downloads_id = 1620866179): ``` https://www.facebook.com/zuck?hc_ref=ARTIabbsED4833Xhh0R0ynoiQkRaK5MjwDzGp8AaJIyir_RMwcu9K2R-TqXg3tZzkGo&fref=nf&pnref=story ``` manage...	push	05 Oct 2018 02:12PM UTC	pypt	travis-ci	pending completion

berkmancenter / mediacloud
62%
master: 70%

berkmancenter / mediacloud 62% master: 70%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

CHANGE BRANCH x

Relevant lines Covered

Source Files on extractor_strip_control_chars

Recent builds

berkmancenter / mediacloud
62%
master: 70%

README BADGES
x

CHANGE BRANCH
x