MITLibraries / timdex-dataset-api / 20069786007

94%

main: 93%

Build Type

Pull #179

github

Committed by ghukill

Commit Message

Join embeddings queries on record metadata

Why these changes are being introduced:

Two nice-to-have functionalities were missing from the first pass of embeddings
reading:

1. filter by record metadata columns now in embeddings schema
2. retrieve record metadata columns in embeddings read methods

There was a deliberate choice to keep embeddings read methods simple in the first
pass.  This builds on that work.

How this addresses that need:

For TIMDEXEmbeddings.read_batches_iter(), the base read method for all embeddings
read methods, perform a join to record metadata via the composite key (timdex_record_id,
run_id, run_record_offset).  Given that record metadata is very fast and memory safe, this
join is too.  By performing this join, we can expose record metadata columns that
intentionally don't exist in the embeddings schema -- e.g. 'source' or 'run_timestamp' --
for filtering and selecting.

Side effects of this change:
* Read methods for TIMDEXEmbeddings can filter and return columns only found in
records metadata tables/views.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/USE-143

Pull Request Pull Request #179: Join embeddings queries on record metadata

Run Details

33 of 33 new or added lines in 2 files covered. (100.0%)

3 existing lines in 1 file now uncovered.

688 of 733 relevant lines covered (93.86%)

0.94 hits per line