snowplow / schema-ddl / 12007810384

81%

master: 81%

Build Type

push

github

Committed by web-flow

Commit Message

Preserve original field order when merging parquet Fields (#213)

We use the `Migrations.mergeSchema` function in the loaders, for
combining a series of schemas (e.g. `1-0-0`, `1-0-1`, `1-0-2`) into a
reconciled column

Before this PR, `mergeSchemas` contained some logic to preserve field
order... but it was the wrong logic. It preserved field order of the
newer schema, ignoring whether a field was present in the older schema.

After this PR, we preserve field order of the older schema.  New fields
added in a later schema are appended to the end of the field list.

This feature change is needed for loaders that only allow field
additions when they are appended to the end of a struct. E.g. the Lake
Loader for Hudi/Glue.

Run Details

15 of 15 new or added lines in 2 files covered. (100.0%)

14 existing lines in 10 files now uncovered.

1277 of 1590 relevant lines covered (80.31%)

0.8 hits per line