MITLibraries / timdex-dataset-api / 15325559881

96%

main: 93%

Pull #146

github

Add run_timestamp column to dataset

Why these changes are being introduced:

During work on yielding only "current" records from the dataset,
where ordering of the ETL runs in the dataset is critical, it
was determined that more time granularity was needed for each
ETL run.

Currently we store the YYYY-MM-DD for each run, but if multiple
runs occur on the same day, we are unable to order them more
granularly.

How this addresses that need:
* Adds new run_timestamp to parquet dataset schema
* Timestamp is minted before any runs are written, and then
used for each row in the ETL run

Side effects of this change:
* All TIMDEX components that use this library for reading
and writing will need a terraform rebuild to pick
up this change.  Otherwise, they need no further modification.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-496

7 of 7 new or added lines in 2 files covered. (100.0%)

300 of 314 relevant lines covered (95.54%)

0.96 hits per line