supabase / etl / 41674793996

81%

Build Type

push

github

Committed by web-flow

Commit Message

fix(clickhouse): increase leniency of retries on proxy 5xx errors and startup probe (#1200)

* fix(clickhouse): retry proxy 5xx error bodies

The clickhouse crate keeps the HTTP status only when an error body is
empty. A proxy that answers with its own body, such as a bare
"Bad Gateway" or an nginx error page, therefore produced a BadResponse
with neither a ClickHouse code nor a status. It was classified as
permanent and stopped replication for manual retry.

Recognize the server errors proxies commonly report in their own body,
as a bare reason phrase or a status line in an error page, and treat
them like the bare 5xx status. Redaction keeps that status, so errors
read "502 Response body omitted" instead of an opaque
"Response body omitted".

* fix(clickhouse): retry startup version probe

The engine support check runs one SELECT version() at startup under the
12 second connectivity budget. A timeout or proxy 5xx failed startup at
once, and replicators retried only through pod restarts. Prod saw this
across several pipelines in the same minutes.

Retry retryable probe failures with the shared backoff helper: four
retries with 2, 4, 8, and 16 second delays. A destination that never
answers now fails startup after up to about 90 seconds. Permanent
failures, such as a rejected login, still fail on the first attempt.
A shutdown signal still cancels startup, including the backoff sleep.

Integration tests put a proxy in front of the test ClickHouse: startup
recovers after a 502 body and fails at once on a 403 body.

Coverage Stats

34 of 35 new or added lines in 2 files covered. (97.14%)

16 existing lines in 5 files now uncovered.

60758 of 75399 relevant lines covered (80.58%)

29960.4 hits per line