Logflare / logflare / eb9e1d34c0c45ef01c0e204fee72286696e541cc

81%

Build Type

push

github

Committed by web-flow

Commit Message

fix(clickhouse): trip circuit breaker immediately on TOO_MANY_PARTS (#3711)

When ClickHouse returns error 252 (TOO_MANY_PARTS), a partition already holds
more active parts than `parts_to_throw_insert`. Retrying inserts in that state
creates yet more parts and worsens the condition. Previously the HTTP client
retried the rejected request and the pipeline accumulated the final response
toward the breaker's 20-failures/30s threshold while requeued events kept
hammering the throttled cluster.

- Do not retry `TOO_MANY_PARTS` responses in the HTTP middleware.
- Add `CircuitBreaker.trip/1` to open the breaker immediately, bypassing the
  failure-count threshold; record the open reason in logs and telemetry.
- Route insert failures by shared ClickHouse error classification so code 252
  trips immediately while other errors retain failure counting.
- Cover the current generation-pointer pipeline and the no-HTTP-retry path.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Coverage Stats

19 of 21 new or added lines in 3 files covered. (90.48%)

3 existing lines in 2 files now uncovered.

14177 of 17514 relevant lines covered (80.95%)

11524.8 hits per line