ukeeper / ukeeper-readability / 24303853688

78%

master: 77%

Build Type

Pull #73

github

Committed by paskal

Commit Message

feat: add per-rule and global Cloudflare routing, 429 retries

HTTP retriever stays the default. Cloudflare is now opt-in at two levels:

- per-rule: new Rule.UseCloudflare field (checkbox in rule editor UI)
  routes requests for that domain through Cloudflare Browser Rendering
- global: --cf-route-all / CF_ROUTE_ALL flag (default false) routes every
  request through Cloudflare

UReadability.pickRetriever(rule) picks: CFRouteAll > rule.UseCloudflare >
default HTTP. extractWithRules now resolves the rule once upfront and
shares it between routing and getContent (was looked up twice).

CloudflareRetriever retries on HTTP 429 with exponential backoff (base 11s,
max 2 retries by default → worst-case 33s of backoff), honours Retry-After
header, and aborts immediately on caller context cancel. MaxRetries=-1
disables retries.

Added WriteTimeout=150s on the HTTP server — was previously unset, allowing
handlers to run forever. 150s covers the worst-case CF path (up to ~123s).

Pull Request Pull Request #73: Modularise URL retrieval with Cloudflare Browser Rendering support

Coverage Stats

206 of 248 new or added lines in 5 files covered. (83.06%)

1 existing line in 1 file now uncovered.

766 of 982 relevant lines covered (78.0%)

9.36 hits per line