Logflare / logflare / 36426890158-2

83%

main: 83%

Build Type

Pull #4049

github

Committed by Baishan

Commit Message

fix(bigquery): key schema-update sampling to BigQuery's own processing rate

process_data/3 sampled schema-update checks against RateSampler's
dispatch-rate key -- the same one maybe_broadcast_and_route/2 bumps on
ingress. That diverges hard from BigQuery's actual processing
throughput during a backlog: draining 10,000 queued events at low
recorded ingress produced near-100% sampling (probability = 1/rate,
and rate stays pinned near the stale ingress figure), running the
schema diff (which happens before Schema.update's own update-time
throttle) on nearly every event instead of a handful. The inverse also
happens -- a stale high rate can suppress checks for an entire slow
drain after ingress stops, since nothing ages it down.

Gives BigQuery its own {source.token, :bq_schema_check} rate, bumped
once per processed batch in handle_batch/4 (including the current
batch, so a large backlog is reflected immediately) and sampled
against in process_data/3 instead of the dispatch-rate key. A source
with no schema row yet always runs its check regardless of rate, so a
large first batch's seeded rate can't skip initial schema discovery.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Pull Request Pull Request #4049: feat: Unify broadcast and schema-update rate gating into one local sampler

Coverage Stats

26 of 26 new or added lines in 3 files covered. (100.0%)

2 existing lines in 2 files now uncovered.

15740 of 18897 relevant lines covered (83.29%)

2029.05 hits per line