supabase / etl / 39294970188

81%

Build Type

push

github

Committed by web-flow

Commit Message

fix(ducklake): retry the right DuckDB errors, redact credentials (#1125)

* ducklake: classify DuckDB errors by type

DuckLake treated every DuckDB error as permanent, and its write, copy,
and inlining paths overrode that with retryable_until_classified, so a
rejected statement was retried as long as a lost connection.

duckdb-rs drops the exception type DuckDB records for a failure, but
DuckDB starts every message with it ("Catalog Error: ..."). DuckLake
now parses that prefix into DuckDbErrorType, mirroring DuckDB's
EXCEPTION_MAP. Transaction conflicts, I/O, HTTP, connection,
interrupt, and out-of-memory errors are retryable; the rest, and
messages without a known type, are permanent. A test pins the format
against the bundled DuckDB.

retryable_until_classified is gone. Write batches and copy inlining
now follow their cause: a missing table or column, a constraint
violation, or a rejected statement stops for an operator right away
instead of after local and pipeline retries. Sanitized DELETE and
UPDATE failures keep the DuckDB type, and report it, without the
message. An invalidated copy buffer stays explicitly retryable.

Blocking-operation deadlines now report DestinationTimeout, which is
retryable, instead of DestinationQueryFailed, which only the removed
helper made retryable on writes. DDL and lookup timeouts now retry.

Part of PIPE-1207.

* ducklake: redact catalog password in DuckDB errors

DuckDB's Postgres extension writes the catalog connection string,
password included, into errors such as a lost catalog connection.
DuckLake attached those errors as sources and logged them, so the
password reached logs, error chains, and table error state. Only the
attach setup step was redacted.

DuckDB errors now become RedactedDuckDbError, in etl_maintenance's
ducklake module, before they are logged or attached. It keeps only
the redacted message and no source, so walking the error chain cannot
reach the original text. In etl-destinations only the re... (continued)

Coverage Stats

425 of 521 new or added lines in 7 files covered. (81.57%)

7 existing lines in 4 files now uncovered.

52551 of 66148 relevant lines covered (79.44%)

10436.35 hits per line