snejus / beetcamp

91%

main: 91%

push

github

fix: use httpx instead of requests to avoid Bandcamp blocking

When using requests/urllib3, Bandcamp response to all requests with
403 errors.

Investigating why, I tried:
- using curl to send the same request: it worked
- writing a tiny Python script to `GET bandcamp.com/` with requests: it
  failed with 403
- waiting a week to see if it solved itself: no luck
- changing the above mentioned script to use http.client or httpx worked

I think that in this case, Bandcamp's Web Application Firewall (WAF)
blocks the requests based not on their contents but on an artifact of
how urllib3 builds/sends the data, since curl with exact same headers
works.

Instead of trying to identify the exact reason, which is quite hard
without any info on Bandcamp's WAF, and fix/workaround that, I rewrote
the very little required HTTP code to use httpx and sidestep the issue.

532 of 587 branches covered (90.63%)

Branch coverage included in aggregate %.

13 of 18 new or added lines in 3 files covered. (72.22%)

1018 of 1114 relevant lines covered (91.38%)

13.59 hits per line