blevesearch / bleve / 23662237690

53%

Build Type

push

github

Committed by web-flow

Commit Message

MB-62182: Avoid re-training vector indexes during merge (#2204)

- The main purpose of this PR is to avoid unnecessary re-training of the
vector indexes during merge process.
- Going by the numbers, we need roughly 156K vectors for a 1M dataset
((min_num_vectors_per_centroid) * num_centroids = 39 * 4 * sqrt(1M)) [as
per
recommendation](https://github.com/facebookresearch/faiss/wiki/FAQ#questions-about-training)
- The data ingestion is now split into 2 phases - the first phase
involves creating a centroid index using the Train() API and the bolt is
recorded with the progress in terms of samples trained upon. The second
phase is just the normal indexing of data using the Batch() or the
Index() APIs.
- Later on, when the vector indexes are getting merged the merger will
use the centroid index to merge the inverted lists (centroids) in a
block-wise fashion without reconstructing the layout.
- The feature can be enabled by passing a `"vector_index_fast_merge":
"true"` key-value pair as part of kvconfig while creating/opening the
index.

Run Details

18 of 87 new or added lines in 7 files covered. (20.69%)

8 existing lines in 1 file now uncovered.

19219 of 36531 relevant lines covered (52.61%)

0.59 hits per line