MITLibraries / timdex-dataset-api / 15169783189

95%

main: 94%

Build Type

Pull #144

github

Committed by ghukill

Commit Message

TIMDEXDataset capable of yielding current records only

Why these changes are being introduced:

With TIMDEXDataset capable of limiting to only parquet files
associated with current runs, the next logical step is providing
the ability to yield only the current version of a record.

This would support a "full refresh" of a TIMDEX source where an
application like TIM could yield only current records for a given
source and index those to Opensearch.

How this addresses that need:

When TIMDEXDataset is loaded with current_records=True, the private
attribute TIMDEXDataset._dedupe_on_read is set to True, informing
any read methods to dedupe during yielding.  Because all read
methods TIMDEXDataset.read_batches_iter() at the lowest level,
the deduping logic is required only there.

Because the ordering of the parquet files is already handled by
the load method, the read methods can be confident they are always
seeing the most recent version of a record first, and thus can
just maintain a "seen" list as they are encountered.  This keeps
the deduplication effectively instant and memory safe; no large
in-memory reordering or deduplication is required.

Side effects of this change:
* Applications like TIM now have the option of yielding only current
records for a source, or all sources, supporting new functionality
like fully reindexing a source in Opensearch from parquet dataset
data alone.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-494

Pull Request Pull Request #144: TIMX 494 - yield deduped, most recent records

Run Details

44 of 45 new or added lines in 3 files covered. (97.78%)

2 existing lines in 1 file now uncovered.

299 of 315 relevant lines covered (94.92%)

0.95 hits per line