blevesearch / bleve / 28504179486

52%

Build Type

push

github

Committed by web-flow

Commit Message

MB-70770: Do not exclude segments from persistence (#2363)

- In the current batch flush architecture, we employ a fixed number of
persister workers each operating on a fixed number of segments based on
the total size of the batch.
- Current algorithm:

> **Inputs:** $N$ unpersisted segments, $P$ persister workers, $M$ max
bytes per batch.
>   
> **Partition:** Greedily assign segments to $P$ batches such that
$dataSize(batch_i) \leq M$,
> yielding batches with $K_1, K_2, \ldots, K_P$ segments where
$\sum_{i=1}^{P} K_i \leq N$.
>    
> **Merge:** Concurrently merge each batch, producing $P$ merged
segments.
>    
>    **Persist:** Persist the $P$ merged segments; exclude the remaining
>    $N - \sum_{i=1}^{P} K_i$ segments from this persistence round.

- This breaks an invariant as we will persist the index snapshot which
has the VB-SeqMax mapping of the unpersisted segments as well, leading
to data loss if the index is reopened.
- An attempt to fix this issue was done in
https://github.com/blevesearch/bleve/pull/2296, but this was not a
complete fix, as it would completely erase the VB-SeqMax mapping from
the persisted bolt snapshot based on the excluded unpersisted segment's
vbucket, leading to bolt corruption, as we merge batches before
dispatching them to scorch.
- It must be noted that the index snapshot stores the max sequence
number for each vbucket based on the overall set of segment snapshots,
allowing us to have multiple batches contributing to the same vbucket.
- This PR, first off, reverts
https://github.com/blevesearch/bleve/pull/2296.
- Second, we introduce a new algorithm for batch flush.

> **Inputs:** $N$ unpersisted segments, $P$ persister workers, $M$ max
bytes per batch.
>
> **Partition:** Greedily assign all $N$ segments into $Q$ batches such
that $dataSize(batch_i) \leq M$,
> yielding batches with $K_1, K_2, \ldots, K_{Q}$ segments where
$\sum_{i=1}^{Q} K_i = N$
>
> **Merge:** Concurrently merge all $Q$ batches using a semaphore ... (continued)

Coverage Stats

179 of 223 new or added lines in 6 files covered. (80.27%)

15 existing lines in 3 files now uncovered.

19716 of 37812 relevant lines covered (52.14%)

0.59 hits per line