supabase / etl / 39296640087

81%

Build Type

push

github

Committed by web-flow

Commit Message

test: add fuzz and property tests for data-loss paths (#1131)

* fix(etl-postgres): decode negative numeric scales

Postgres 15+ stores the NUMERIC scale as an 11-bit two's-complement
field, but numeric_modifiers read 16 bits, so numeric(5,-2) decoded
as scale 2046. Mirror numeric.c and treat any typmod below VARHDRSZ as
unconstrained.

The old unit tests built typmods with the same wrong encoding, so they
passed on the bug. Replace them with atttypmod values read from a PG 18
server and an exhaustive check against the Postgres encoder for every
valid precision and scale. DuckLake still maps negative scales to
varchar, so its output does not change.

* test(etl): fuzz pgoutput tuple decoding

Add the decode_pgoutput_tuple cargo-fuzz target. It builds a small
schema and replica-identity masks from the input, assembles a pgoutput
Insert, Update, or Delete body, parses it with postgres_replication,
and runs the etl decoders. Decoding must never panic, and accepted
rows must have the replicated width with sorted, in-range, disjoint
missing-column indexes.

Six hand-written seeds cover insert, unchanged TOAST, key change, full
old row, and key-only and full deletes. The target joins the nightly
Fuzz matrix; the PR corpus replay finds it through cargo fuzz list.
A 60 second local run made 311,000 executions with no findings.

* test: add property tests for data-loss paths

Add model and differential properties on the shared budget runner for
the paths where a bug would silently corrupt or lose replicated data:

- etl: pgoutput update and delete decoding against a reference model,
  RELATION materialization, schema-change plan replay, durable table
  state, COPY vs pgoutput decoding, JSON and array element laws, and
  default expressions against the value Postgres backfills.
- BigQuery: CDC merge order and protobuf wire agreement.
- ClickHouse: CDC history with crash and replay, and SQL quoting.
- DuckLake: encoder agreement, identity predicates, and replay.... (continued)

Coverage Stats

3273 of 3719 new or added lines in 14 files covered. (88.01%)

180 existing lines in 5 files now uncovered.

56036 of 69914 relevant lines covered (80.15%)

43044.9 hits per line