MITLibraries / timdex-index-manager / 21404761219

96%

main: 96%

Build Type

Pull #377

github

Committed by ghukill

Commit Message

Bulk update source with current source embeddings

Why these changes are being introduced:

The first pass at bulk updating pre-existing documents, encapsulated in the
command `bulk-update-embeddings` required passing a `--run-id` to target a specific
ETL run.  This aligns with the most common use case of indexing embeddings within
an ETL run.

However, we have use cases now for indexing all current embeddings for a given source
into Opensearch.  These current embeddings may span multiple ETL runs.

How this addresses that need:

Updates the `bulk-update-embeddings` CLI command to require only `--source`,
defaulting to retrieving all current embeddings for that source.  This logic is
identical to what `reindex-source` was already doing, but is decoupled from
re-indexing the documents themselves which is not always required.

While working on this, it was decided that raising an exception for a missing
document when performing updates is not ideal.  Some sources have indexing issues,
and we have historically skipped those records.  When we get to bulk updates, it's
possible that we have embeddings for documents that were never indexed; we will log
and skip them now in a similar fashion.

Side effects of this change:
* CLI supports ad-hoc indexing of all current embeddings for a source

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/USE-355

Pull Request Pull Request #377: USE 355 - index current embeddings for a source

Run Details

6 of 7 new or added lines in 2 files covered. (85.71%)

1 existing line in 1 file now uncovered.

476 of 496 relevant lines covered (95.97%)

0.96 hits per line