brawer / osmdiffs / 40201287415

94%

push

github

feat: compute QRank quantiles for brand Wikidata IDs (#848)

* feat: compute QRank quantiles for brand Wikidata IDs

Fetch the latest QRank file as a third pipeline input, discovered
through QRank's datapackage.json and checked against the SHA-256
listed there. A new step, compute_qrank_quantiles, gives each ID in
wikidata-ids its QRank quantile among those same IDs and writes
them to qrank-quantiles, a BlobTable of little-endian f32 values.

Quantiles are mid-rank, so ties (most notably the IDs without any
pageviews) share the middle of their range. Nothing consumes the
quantiles yet; they are a first ranking signal for #713.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Nw5ZNMD34EwRUnASRv4jy

* fix: send a User-Agent with HTTP requests

Toolforge, where QRank is hosted, answers 403 Forbidden to requests
without one, as Wikimedia's User-Agent policy says it may.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Nw5ZNMD34EwRUnASRv4jy

* fix: identify as OSMDiffsBot in the User-Agent

Wikimedia's User-Agent policy asks automated clients to say they are
bots.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Nw5ZNMD34EwRUnASRv4jy

---------

Co-authored-by: Claude <noreply@anthropic.com>

393 of 398 new or added lines in 5 files covered. (98.74%)

12348 of 13149 relevant lines covered (93.91%)

1342.14 hits per line