MITLibraries / timdex-dataset-api / 18600752044

93%

main: 93%

push

github

Lazy load current records

Why these changes are being introduced:

Current TIMDEX records has been a consistent source of complexity
and performance concerns.  And, one of the defining features of TDA, so
it's worth getting right.

The previous approach was to materialize lightweight metadata records
about all current versions of each record in memory as a DuckDB temp table.
This made repeated read queries pulling only current records more efficient,
but was unnecessarily loading that data into memory for operations like
writing data or reading a specific run (not current record associated).

It turns out that reading current records is somewhat rare, and when it
does happen, it's usually a one-off request as part of a larger operation
like re-indexing a source in TIM.

How this addresses that need:

The new approach is a hybrid between a view only (lazy evaluation) and
a temporary table in memory (eager evaluation).

By default, a view is created, which either a) does not get used or b)
is used but only 1-2 times per session and the lazy evaluation of a view
is okay.

Alternatively, TIMDEXDataset can be initialized with 'preload_current_records=True'
if it's known that a multiple requests for current records will be needed in the
session and it's worth the time and memory hit upfront.

Side effects of this change:
* For most operations in the TIMDEX ETL pipeline, which don't use current records,
the load time and memory usage is fairly dramatically decreased.

Relevant ticket(s):
- https://mitlibraries.atlassian.net/browse/USE-58
  - prep work for new methods

9 of 9 new or added lines in 2 files covered. (100.0%)

525 of 564 relevant lines covered (93.09%)

0.93 hits per line