MITLibraries / timdex-dataset-api / 15169020046

95%

main: 93%

push

github

Extend load to optionally limit to current parquet files

Why these changes are being introduced:

With the creation of TIMDEXRunManager we now have the ability to
identify parquet files associated with current ETL runs for all
or a given source.  This could be used to limit the TIMDEXDataset
on load to only read from those parquet files.

How this addresses that need:
* Updates TIMDEXDataset.load() with a new 'current_records' flag that
if True will use TIMDEXRunManager to get a list of parquet files to
upload the dataset paths with.

Side effects of this change:
* None without explicit use.  Eventually, could be utilized by
contexts where only parquet files associated with current ETL
runs are needed.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-494

22 of 22 new or added lines in 2 files covered. (100.0%)

280 of 295 relevant lines covered (94.92%)

0.95 hits per line