OCHA-DAP / hdx-resource-changedetection / 23779230115

71%

main: 71%

Pull #9

github

Refactor concurrency and rate limiting

Reworks the asyncio concurrency and rate limiting logic to improve performance and stability.

- Replaces `limit_per_host` with a more granular system using per-host semaphores.
- Introduces different rate limits for HDX filestore hosts (10/sec) versus other hosts (4/sec).
- Adds a global connection limit and an event loop task semaphore to prevent overwhelming the system.
- A new chunking `list_distribute_contents` utility to support better task scheduling.
- Reduces the in-memory zip processing threshold to 75MB.

101 of 171 new or added lines in 5 files covered. (59.06%)

498 of 698 relevant lines covered (71.35%)

0.71 hits per line