MITLibraries / timdex-dataset-api / 12282750069

99%

main: 93%

Build Type

Pull #15

github

Committed by jonavellecuerdo

Commit Message

Update TIMDEXDatase.write method to only overwrite similarly named parquet files

Why these changes are being introduced:
* Since the TIMDEXDataset partitions are now the [year, month, day]
of the 'run_date', parquet files from different source runs
will be written to the same partition. The previous configuration
of existing_data_behavior="delete_matching" would result in
the deletion of any existing parquet files from the partition directory
with every source run, which is not the desired outcome.
To support the new partitions, this updates the configuration
existing_data_behavior="overwrite_or_ignore" which will
ignore any existing data and will only overwrite files with the
same filename.

How this addresses that need:
* Set existing_data_behavior="overwrite_or_ignore" in ds.write_dataset method call
* Add unit tests to demonstrate updated existing_data_behavior

Side effects of this change:
* In the event the multiple runs are performed for the same 'source' and 'run-date',
which is unlikely to occur, parquet files from both runs will exist in the
partitioned directory. DatasetRecords are can still be uniquely identified via the
'run_id' column.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-432

Pull Request Pull Request #15: Rework dataset partitions to only year, month, day

Run Details

21 of 21 new or added lines in 2 files covered. (100.0%)

1 existing line in 1 file now uncovered.

148 of 151 relevant lines covered (98.01%)

0.98 hits per line