MITLibraries / timdex-dataset-api / 12243904605

99%

main: 93%

Build Type

push

github

Committed by jonavellecuerdo

Commit Message

Rework dataset partitions to only year, month, day

Why these changes are being introduced:
* These changes simplify the partitioning schema for the TIMDEXDataset,
allowing the app to take advantage of PyArrow's memory-efficient
processes for reading and writing Parquet datasets. Furthermore, the
new partitioning schema will result in a more efficient, coherent
folder structure when writing datasets. For more details, see:
https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/4094296066/Engineering+Plan+Parquet+Datasets+for+TIMDEX+ETL#Rework-Dataset-Partitions-to-use-only-Year-%2F-Month-%2F-Day.

How this addresses that need:
* Update TIMDEX_DATASET_SCHEMA to include [year, month, day]
* Update DatasetRecord attrs to include [year, month, day] and
  set [source, run_date, run_type, run_id, action] as primary columns
* Add post_init method to DatasetRecord to derive partition values
  from 'run-date
* Remove 'partition' values from DatasetRecord.to_dict
* Remove 'partition_values' mixin from TIMDEXDataset.write to reduce
  complexity and have write method utilize DatasetRecord partition
  columns instead.
* Update unit tests to use new partitions and remove deprecated tests

Side effects of this change:
* The new partitioning schema introduces a 3-level folder structure
within TIMDEXDataset.location (i.e. the base path of the dataset)
for [year, month, day], where the leaf node will contain parquet files
for every source run.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-432

Run Details

18 of 18 new or added lines in 1 file covered. (100.0%)

1 existing line in 1 file now uncovered.

148 of 151 relevant lines covered (98.01%)

0.98 hits per line