snowplow / schema-ddl

80%

master: 81%

LAST BUILD ON BRANCH refs/tags/0.26.0-M1

branch: refs/tags/0.26.0-M1

CHANGE BRANCH

x

Reset

Sync Branches

• refs/tags/0.26.0-M1
• PDP-1396
• develop
• feature/enum-zstd
• feature/parquet-no-intermediate
• fix-ci
• fix-develop
• fix-ipv6
• fix-recovery-table-name
• fix/same-type-incompat
• improve-migration-signature
• improve-migration-signature-amendment
• master
• max-json-depth
• migration-field-order
• omit-empty-struct
• refs/pull/193/merge
• refs/pull/195/merge
• refs/tags/0.16.0
• refs/tags/0.16.0-M3
• refs/tags/0.16.0-rc14
• refs/tags/0.16.0-rc15
• refs/tags/0.16.0-rc16
• refs/tags/0.16.0-rc17
• refs/tags/0.16.0.-rc16
• refs/tags/0.16.1-M1
• refs/tags/0.16.1-M2
• refs/tags/0.17.0
• refs/tags/0.17.0-M1
• refs/tags/0.17.0-M2
• refs/tags/0.17.1
• refs/tags/0.17.2-M1
• refs/tags/0.18.0
• refs/tags/0.18.0-M10
• refs/tags/0.18.0-M11
• refs/tags/0.18.0-M12
• refs/tags/0.18.0-M13
• refs/tags/0.18.0-M14
• refs/tags/0.18.0-M15
• refs/tags/0.18.0-M16
• refs/tags/0.18.0-M17
• refs/tags/0.18.0-M18
• refs/tags/0.18.0-M6
• refs/tags/0.18.0-M7
• refs/tags/0.18.0-M8
• refs/tags/0.18.0-M9
• refs/tags/0.18.1
• refs/tags/0.18.1-M4
• refs/tags/0.18.2
• refs/tags/0.19.0
• refs/tags/0.19.0-M1
• refs/tags/0.19.3
• refs/tags/0.19.3-M1
• refs/tags/0.19.3-M2
• refs/tags/0.19.3-M3
• refs/tags/0.19.4
• refs/tags/0.20.0
• refs/tags/0.20.0-M1
• refs/tags/0.21.0
• refs/tags/0.21.0-M1
• refs/tags/0.22.0
• refs/tags/0.22.1
• refs/tags/0.22.2
• refs/tags/0.23.0
• refs/tags/0.23.0-M1
• refs/tags/0.24.0
• refs/tags/0.24.0-M1
• refs/tags/0.24.0-M2
• refs/tags/0.25.0
• refs/tags/0.25.0-M1
• refs/tags/0.26.0
• refs/tags/0.26.0-M2
• refs/tags/0.27.0
• refs/tags/0.27.0-M1
• release/0.19.3
• release/0.19.4
• release/0.21.0
• release/0.22.0
• release/0.22.1
• release/0.22.3
• release/0.23.0
• release/0.24.0
• release/0.25.0
• release/0.27.0
• upgrade_dependencies

Build # 11860212877

Build Type

Pull #213

github

Committed by istreeter

Commit Message

Preserve original field order when merging parquet Fields

We use the `Migrations.mergeSchema` function in the loaders, for
combining a series of schemas (e.g. `1-0-0`, `1-0-1`, `1-0-2`) into a
reconciled column

Before this PR, `mergeSchemas` contained some logic to preserve field
order... but it was the wrong logic. It preserved field order of the
newer schema, ignoring whether a field was present in the older schema.

After this PR, we preserve field order of the older schema.  New fields
added in a later schema are appended to the end of the field list.

This feature change is needed for loaders that only allow field
additions when they are appended to the end of a struct. E.g. the Lake
Loader for Hudi/Glue.

Pull Request Pull Request #213: Preserve original field order when merging parquet Fields

Run Details

12 of 12 new or added lines in 2 files covered. (100.0%)

14 existing lines in 10 files now uncovered.

1276 of 1589 relevant lines covered (80.3%)

0.8 hits per line