MITLibraries / timdex-index-manager / 20348552949

push

github

Use metadata in 'bulk-update-embeddings' (#375)

Why these changes are being introduced:
* With TDA 3.8.0, we can now retrieve record metadata columns in
embeddings read methods. Filtering embeddings by `action="index"`
prevents any attempt to update documents that do not exist
in OpenSearch (`action="delete"`), which results in an API error..
This is important especially with the current state of tim.opensearch.bulk_update,
which will raise a BulkOperationError and cause the 'bulk_update_embeddings'
CLI command to exit early.

This also includes an additional change to also index embeddings
when performing a reindex.

How this addresses that need:
* Filter embeddings by action="index"
* Install latest version of timdex-dataset-api (latest commit)
* Update embeddings in fixtures/test/dataset to use 'embeddings_timestamp"

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/USE-273

7 of 10 new or added lines in 1 file covered. (70.0%)

471 of 492 relevant lines covered (95.73%)

0.96 hits per line