MITLibraries / timdex-dataset-api / 12243134456

99%

main: 93%

Build Type

push

github

Committed by jonavellecuerdo

Commit Message

Add post_init method to DatasetRecord to derive partition values

Why these changes are being introduced:
* The [year, month, day] partition values can be derived from
DatasetRecord.run_date and removing the 'partition_values'
mixin from TIMDEXDataset.write method reduces the
complexity of the function by making the write method directly
pull partitions from the DatasetRecord partition columns
instead. For more details, see:
https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/4094296066/Engineering+Plan+Parquet+Datasets+for+TIMDEX+ETL#Rework-Dataset-Partitions-to-use-only-Year-%2F-Month-%2F-Day.

How this addresses that need:
* Derive [year, month, day] attrs for DatasetRecord from 'run-date'
to use as partition values
* Remove option 'partition_values' from TIMDEXDataset.write method
and rely on valid 'run-date' values to derive partitions,
raising an error if 'run-date' cannot be parsed
* Remove deprecated tests

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-432

Run Details

10 of 10 new or added lines in 1 file covered. (100.0%)

2 existing lines in 2 files now uncovered.

148 of 151 relevant lines covered (98.01%)

0.98 hits per line