MITLibraries / timdex-dataset-api / 16751506229

94%

main: 93%

Pull #158

github

Write append deltas on ETL records data write

Why these changes are being introduced:

With the new metadata approach, an important component are
"append deltas".  These are standalone parquet files that contain
metadata about the records added to the ETL records data parquet
files.

These eventually are merged into the main static metadata file,
but until then are needed for metadata queries.

How this addresses that need:

During TIMDEXDataset.write(), after each ETL parquet file is
written we lean on new method TIMDEXDatasetMetadata.write_append_delta_duckdb()
to read metadata from that file and write a new append delta parquet
file.

This is performed entirely in a DuckDB context, allowing for simple
column selection.

Side effects of this change:
* During dataset write, append deltas will be created.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-527

64 of 68 new or added lines in 2 files covered. (94.12%)

385 of 410 relevant lines covered (93.9%)

0.94 hits per line