supabase / etl / 26146978700

73%

Build Type

push

github

Committed by web-flow

Commit Message

Add ClickHouse ReplacingMergeTree support (#750)

* feat(clickhouse): thread engine through destination config

Adds `ClickHouseEngine` (`merge_tree` | `replacing_merge_tree`) in
`etl-config::shared`, defaulting to `ReplacingMergeTree`. Wires the
field through `DestinationConfig::ClickHouse`,
`DestinationConfigWithoutSecrets::ClickHouse`,
`ClickHouseInserterConfig`, and the replicator init path.

No behavioral change: the destination still emits `MergeTree` DDL.
Subsequent commits will branch DDL, encoding, and validation on the
engine.

* feat(clickhouse): RMT DDL builders and `__current` view

Adds `create_replacing_merge_tree_sql` (PK-required, ordered by
`primary_key_ordinal_position`, trailing `_etl_lsn` + `_etl_deleted`)
and `create_current_view_sql` (FINAL + tombstone filter, exposes only
user columns). Renames `build_create_table_sql` to
`create_merge_tree_sql` and adds a `create_table_sql` dispatcher.

Wires `create_table_with_metadata` and `recover_applying_metadata`
through the dispatcher and emits the companion view under RMT.
`expected_clickhouse_column_names` is now engine-aware via
`trailing_cdc_column_names`.

Adds `DdlKind::CreateView` for telemetry. No row-encoding changes yet;
RMT tables cannot be written end-to-end until the next commit.

* ref(clickhouse): polish RMT DDL helpers from review

- Rename `issue_create_table_ddl` to `issue_create_table_stmt`.
- Reword CDC/etl column constant doc-comments to lead with the engine.
- Simplify `expected_clickhouse_column_names` to use a chained iterator.
- Add blank lines around the early-return and final `Ok(format!)` in
  `create_replacing_merge_tree_sql`.

* feat(clickhouse): RMT encoder and validation

Row encoding now branches on engine via `append_cdc_columns`:
- MergeTree (unchanged): `cdc_operation` + `cdc_lsn` (commit_lsn).
- ReplacingMergeTree: `_etl_lsn` (start_lsn, the RMT version) + `_etl_deleted`
  tombstone. `start_lsn` is globally monotonic, so multi-event same-commit
  tra... (continued)

Coverage Stats

645 of 682 new or added lines in 10 files covered. (94.57%)

49 existing lines in 5 files now uncovered.

31290 of 41962 relevant lines covered (74.57%)

905.86 hits per line