MITLibraries / timdex-dataset-api / 15121174566

94%

main: 93%

push

github

Add TIMDEXRunManager and TIMDEXSource classes

Why these changes are being introduced:

Until now, this library aimed to support the known use cases
of efficiently writing the results of an ETL run to the dataset
and efficiently reading all records from an ETL run for indexing
into Opensearch.  However, it was known that eventually this library
should support more nuanced access of the dataset, including a
representation of the current state of a source in TIMDEX.

How this addresses that need:

This commit introduces two new classes:
1. TIDMEXRunManager
2. TIMDEXSource

TIMDEXRunManager is a utility class that efficiently retrieves
metadata about all ETL runs in the dataset. Equally important, it is
capable of then returning specific parquet files from the dataset
that are associated with current records in TIMDEX for the source.

Building on this, TIMDEXSource aims to provide the more nuanced behavior
of yielding only the current versions of records for a source.  This class
extends TIMDEXDataset, providing all the base class functionality,
with the layered opinionation of only yielding "current" versions of
a record.

Together, these classes provide efficient and normalized patterns
for yielding all current records from the dataset for a source,
pulling them from the timeseries nature of the dataset.

Side effects of this change:
* None at this time

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-494

112 of 122 new or added lines in 4 files covered. (91.8%)

305 of 324 relevant lines covered (94.14%)

0.94 hits per line