rm-hull / news-barge / 37197501899

93%

push

github

fix: reduce Playwright timeout to 7s + broaden ad/tracker blocking + fix profile instrumentation (#50)

* fix: reduce Playwright page.goto timeout from 30s to 5s

The 30s timeout was being hit almost universally on ad-heavy sites like
Leeds Live (playwright_wait_until: networkidle), which have continuous
background network activity from ads/tracking scripts that prevent
'networkidle' from ever firing.

Profiling from the Actions run showed 206 Playwright calls averaging
32.4s each — nearly all hitting the timeout ceiling — versus 0.62s avg
for aiohttp. This wasted ~6,400 seconds per scrape run.

Reducing to 5s: pages that load DOM content within 5s succeed as before,
while genuinely stuck pages fail fast instead of blocking for 30s.

* fix: broaden ad/tracker blocking + raise timeout to 7s

- Increase PLAYWRIGHT_TIMEOUT 5000→7000ms: pages that succeed load
  in 1–2s; 7s gives CI a 3× buffer for slower network + bot-challenge
  overhead while still failing 4× faster than the original 30s.
- Replace the glob-string route pattern with a function-based matcher
  (_is_blocked_request) that checks the full URL including hostname.
  The old '**/{analytics,...}**' glob only matched keywords in the URL
  *path* and missed domains like cdn.taboola.com, cmp.inmobi.com, etc.
- Add 7 additional ad/tech/tracking keywords to BLOCKED_REQUEST_KEYWORDS
  in constants.py: taboola, dotmetrics, amplitude, googletagmanager,
  viafoura, inmobi, mantis-awx.  These were identified via network
  inspection of Leeds Live and account for ~40% of the 129 total
  requests that don't reach content-serving domains.
- Update test_create_shared_context_registers_analytics_route_once to
  assert callable matcher instead of string pattern; add
  test_blocked_request_matcher_catches_ads_and_tracking covering both
  positive and negative URL cases.

* test: cover slow-fetch warning + move browser_semaphore out of @profiled

- Remove browser_semaphore parameter from fetch_html_playwright()... (continued)

33 of 35 new or added lines in 4 files covered. (94.29%)

767 of 825 relevant lines covered (92.97%)

0.93 hits per line