knyar / nginx-lua-prometheus / 34268581129

98%

master: 97%

Build Type

push

github

Committed by web-flow

Commit Message

Fix histogram bucket consistency during concurrent collection (#179)

Concurrent histogram updates and collection can report observations in ranges
that have never received a value. Store each observation in one disjoint range
and construct the cumulative buckets from those range counters during collection.
This also makes `+Inf` and `_count` identical in every exported label set.

Related to #161, which partially mitigated the race by changing increment order.
The per-worker counter buffer is flushed with `pairs()`, and collection can also
overlap an otherwise complete flush, so increment order alone cannot prevent it.

### Concrete failure

Suppose a histogram has bounds `{1, 10, 100}`, and all ten observations are `0.5`:

| Scrape | `le="1"` | `le="10"` | `le="100"` | `le="+Inf"` |
|---|---:|---:|---:|---:|
| Before another observation | 10 | 10 | 10 | 10 |
| During an overlapping flush | 10 | 11 | 11 | 11 |
| After the flush | 11 | 11 | 11 | 11 |

The collector reads `le="1"` before another worker flushes one more `0.5`
observation, then reads the other buckets afterward. The middle scrape claims
one observation in `(1, 10]`, although every observation was below 1. Other
interleavings can expose decreasing cumulative buckets or disagreeing `+Inf`
and `_count` values.

Consumers that convert cumulative buckets into per-range observations can turn
this discrepancy into phantom slow samples, inflating tail percentiles and
histogram-derived maxima. A later consistent scrape does not necessarily undo
observations already ingested into a downstream distribution.

### Changes

- Increment one disjoint bucket and `_sum` per observation. Collection reads
  each bucket once and builds cumulative counts from those same reads.
- Publish one descriptor per label set and export all configured buckets,
  including zeros, so a newly observed label set has a complete schema.
- Isolate the new internal storage from legacy cumulative cells and different
  bucket lay... (continued)

Coverage Stats

543 of 543 new or added lines in 2 files covered. (100.0%)

22 existing lines in 1 file now uncovered.

1504 of 1537 relevant lines covered (97.85%)

150.18 hits per line