MITLibraries / timdex-dataset-api / 15619230762

95%

main: 93%

push

github

Yield current records by utilizing dataset fragments

Why these changes are being introduced:

A bug was discovered where the yielding of currect records could throw
an error for the unfiltered + filtered batches getting combined
via zip(..., strict=True).  It was learned that despite reading the
same parquet files, the number of batches *can* differ if filtering is
applied.  As such, we cannot just zip together the two batch iterators.

How this addresses that need:

What we can do is "backfill" seen timdex_record_ids everytime we get
to the end of a parquet file, and we can safely identify parquet files
via a dataset fragment,
https://arrow.apache.org/docs/cpp/api/dataset.html#_CPPv4N5arrow7dataset8FragmentE.

The new approach is as follows:
- yield filtered records, noting the fragment index
- if we encounter a new fragment we pause and backfill
timdex_record_ids from unfiltered batches that are part of the previous fragment
- we continue yielding records from the next fragment, confident that even if
filtered out we have marked all records as seen from the previous fragment (
more recent versions)

Side effects of this change:
* None, sans bug fixes for large current record pulls like Alma

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-504

41 of 45 new or added lines in 3 files covered. (91.11%)

333 of 350 relevant lines covered (95.14%)

0.95 hits per line