MITLibraries / timdex-index-manager / 21869162588

96%

main: 96%

Pull #379

github

Remove fulltext exclusion from _source mapping

Why these changes are being introduced:

Originally, we found excluding the newish fulltext field from the _source mapping
in Opensearch was beneficial.  This would keep the fulltext from getting returned
during casual Opensearch queries and/or when an explicit list of fields to return
was not used.

What this exclusion does NOT do -- and this is important to highlight -- is reduce
storage in any meaningful way.  At this time, the way in which we store fulltext
in Opensearch has the same storage footprint if we exclude from _source or not.

That exclusion DID have a negative side effect however.  We later implemented a
TIM command that would update records in Opensearch with embeddings, using
opensearch-py built-in, bulk update methods.

It would appear, though not well documented, that because fulltext was excluded
from the _source fields returned, it was removing that field value when the
document was updated.  Conceptually, it's like Opensearch extracts the document
as represented by _source, interleaves updates, and then reinserts.

How this addresses that need:

Because the `fulltext` exclusion field from _source really had no positive effects,
but was found to have negative ones, electing to remove that exclusion.  This will
result in fulltext getting included in the default _source response.

It is worth noting that in the TIMDEX API -- the largest usage of Opensearch -- we
do explicitly state columns for returning, so it's believed we are not paying the
penalty of transferring the fulltext over the wire each request.  This will be
confirmed and addressed outside the scope of this work.

Side effects of this change:
* fulltext is returned by default as part of _source document
* bulk updating of documents does not remove the fulltext field

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/USE-403

1 of 1 new or added line in 1 file covered. (100.0%)

478 of 500 relevant lines covered (95.6%)

0.96 hits per line