thumbor / remotecv / 39878788232

94%

push

github

Add timeout, body size limit and scheme check to HTTP loader (#116)

load_sync called urlopen with no timeout and read the whole response into
memory, and urllib's default redirect handler follows redirects to ftp as
well as http(s). The URL comes from the job payload, which in the thumbor
integration is the end user's image URL, so a single slow or oversized
origin could pin a worker child indefinitely or exhaust its memory, and a
redirect could point the worker at a scheme nobody intended it to speak.

The loader now opens the URL through an opener whose redirect handler only
accepts http and https, rejects the initial URL when its scheme is anything
else, applies a configurable socket timeout (--loader-timeout, default 20s) and
enforces an optional maximum body size (--loader-max-body-size, checked against
a well-formed Content-Length first and then against the bytes actually read).
Both options refuse at startup the values that would fail every fetch (a
timeout that is zero, negative, NaN or infinite, a negative size). The body
limit defaults to 0 (unlimited) to keep existing deployments working; the
timeout is per socket operation, so DETECTOR_TIMEOUT remains the hard cap for a
slowly dripping origin. This change is stacked on #123 and should merge after
it.

45 of 45 new or added lines in 2 files covered. (100.0%)

543 of 618 relevant lines covered (87.86%)

4.39 hits per line