blevesearch / bleve / 28983703970

52%

Build Type

push

github

Committed by web-flow

Commit Message

MB-72725: Improve efficiency of index compaction (#2373)

- Fixed the planner picking a single-segment ("solo") fully live roster
as the best candidate, which skipped a merge task. We now deliberately
avoid hitting into this degenerate case, and only create merge tasks
that actively reclaim deletes.
- Fixed the file-size budget being computed from raw file size, which is
static and ignores logical deletes. It now uses live file size, so
deleted data no longer inflates the budget.
- Fixed vector-segment eligibility being based on raw file size, so a
4GB vector segment with 99% of its vectors deleted would never become
eligible for compaction. Eligibility now uses live file size.
- Fixed the empty-segment cleanup task being counted toward the budget,
even though merging empty segments produces no output segment. Only real
merges now count toward the budget.
- Overhauled unit tests to assert convergence and the expected segment
layout after the index settles.
- The merger now introduces merged segments in one-shot; it creates one
merged segment per task generated by the planner and introduces all
merged segments in one go. Earlier we were introducing each merged
segment one-by-one, which would reduce concurrency in high-ingest
workloads due to channel contention.
- Increase test coverage by adding unit tests for scorch.

Coverage Stats

195 of 213 new or added lines in 4 files covered. (91.55%)

4 existing lines in 2 files now uncovered.

19841 of 37900 relevant lines covered (52.35%)

0.59 hits per line