rero / rero-ils / 33401872719

91%

push

github

perf(search): right-size the index shards

Measured on a copy of the production cluster: 528 primary shards
for 44.68 GB, an average of 86.6 MB per shard, and 46 of the 70
indices below 10 MB per shard.

`operation_logs` is where that concentrates. Its 35 yearly
indices carry 8 shards each: 560 shards, more than half the
cluster, for 27 GB — and the years before 2021 all sit under
50 MB per shard. Going to 2 removes 426 shards.

Two rather than one because the `stats` module aggregates over
these indices. Measured on `operation_logs-2023`, 21.4 million
entries: 157 ms across 8 shards, and 133 ms for a single shard
covering only 2.7 million of them, so a single shard for the
whole index would take about a second. Those aggregations are
nightly and tolerant, but two shards cost only 35 shards more
than one across the family.

The migration indices asked for 2 replicas, which needs at least
three data nodes to turn green, for indices that hold no
document. Rather than correct the value in place, give them the
template every other index already has: their shard count was
the only one set in Python, as explicit index settings that beat
any template whatever its order.

The new template declares nothing but the two counts, at order
10, so `migrations-*` keeps everything `record.json` gives it at
order 0 — verified on an index built from both: 1 shard and 1
replica from the new template, `max_result_window` still
inherited. That matters because `Migration.name` and
`.description` are `Text` fields, analysed by the `default`
analyzer that `record.json` defines.

`es_templates/v7/record.json` keeps its 8 shards on purpose. The
per-shard aggregation cost is linear, 0.38 ms per thousand
documents examined, and a facet query takes 102 ms across the 8
shards of `documents` against 100 ms on a single shard holding an
eighth of the data: the shards work in parallel and halving them
would double the latency of the most critical query.

Existing indices keep their cur... (continued)

2 of 2 new or added lines in 1 file covered. (100.0%)

24360 of 26648 relevant lines covered (91.41%)

0.91 hits per line