berkmancenter / mediacloud / 5754

62%

master: 70%

Pull #490

travis-ci

Strip non-printable control characters before extracting

A certain download (downloads_id = 1620866179):

```
https://www.facebook.com/zuck?hc_ref=ARTIabbsED4833Xhh0R0ynoiQkRaK5MjwDzGp8AaJIyir_RMwcu9K2R-TqXg3tZzkGo&fref=nf&pnref=story
```

managed to return ~11 MB of data, most of which is weird sequences of
control characters:

```
0004f040: 673e 3c61 2068 7265 663d 2268 7474 7073  g><a href="https
0004f050: 3a2f 2f69 642d 6964 2e66 6163 6562 6f6f  ://id-id.faceboo
0004f060: 6b2e 636f 6d2f 7365 7074 6979 616e 692e  k.com/septiyani.
0004f070: 6265 6c6c 6122 3e4d 6172 6b20 5a75 636b  bella">Mark Zuck
0004f080: 6572 6265 7267 3c2f 613e 3c2f 7374 726f  erberg</a></stro
0004f090: 6e67 3e3c 2f64 6976 3e3c 6469 7620 636c  ng></div><div cl
0004f0a0: 6173 733d 2270 726f 6669 6c65 4672 6965  ass="profileFrie
0004f0b0: 6e64 7354 6578 7420 7072 6f66 696c 6546  ndsText profileF
0004f0c0: 7269 656e 6473 4279 6c69 6e65 2066 736d  riendsByline fsm
0004f0d0: 2066 776e 2066 6367 223e 28e3 8080 e380   fwn fcg">(.....
0004f0e0: 80e3 8080 20e3 8080 e380 80e3 8080 20e3  .... ......... .
0004f0f0: 8080 e380 80e3 8080 20e3 8080 e380 80e3  ........ .......
0004f100: 8080 20e3 8080 e380 80e3 8080 20e3 8080  .. ......... ...
0004f110: e380 80e3 8080 20e3 8080 e380 80e3 8080  ...... .........
0004f120: 20e3 8080 e380 80e3 8080 20e3 8080 e380   ......... .....
0004f130: 80e3 8080 20e3 8080 e380 80e3 8080 20e3  .... ......... .
0004f140: 8080 e380 80e3 8080 20e3 8080 e380 80e3  ........ .......
0004f150: 8080 20e3 8080 e380 80e3 8080 20e3 8080  .. ......... ...
```

(The 0xe3 0x80 0x80 goes on for the rest of the returned file.)

Is this a UTF-8 decoding error on our end? A buffer overflow somewhere?
Dunno.

Readability is not too happy with the control character sequence and got
stuck at it. So:

* Strip ASCII control characters (except for "\n", "\r" and "\t") before
extracting;
* Strip varied whitespace sequences (for example, not only "\t\t\t", but
also "\t\n \t\n " before extracting).

9 of 9 new or added lines in 2 files covered. (100.0%)

14190 of 22752 relevant lines covered (62.37%)

312.27 hits per line