DigitalPebble / storm-crawler / 6381452385

46%

push

github

Improvements and fixes to HttpRobotRulesParser when following redirects (#1103)

- remove unsafe check for absolute URLs on redirects (location header)
- resolve relative redirects using the redirect source as base URL
- cache robot rules for all /robots.txt (if on default location) in
  a chain of redirects

Signed-off-by: Sebastian Nagel <sebastian@commoncrawl.org>

1436 of 3648 branches covered (0.0%)

Branch coverage included in aggregate %.

4428 of 9593 relevant lines covered (46.16%)

1.9 hits per line