nbari / pg_exporter / 34208940068

90%

main: 90%

Build Type

push

github

Committed by nbari

Commit Message

fix: record aborted scrapes as aborted and coalesce overlapping OS samples

Follow-ups from reviewing 006acaa.

Aborted scrapes masqueraded as successes in the exporter's self-metrics.
When a scrape is aborted mid-flight (timeout or client disconnect, #34),
every in-flight collector's ScrapeTimer was dropped without an outcome, and
Drop recorded that as a success with a timeout-sized duration: during exactly
the incident those metrics exist for, last_scrape_success read 1 and
pg_exporter_collector_scrape_duration_seconds gained a bogus +Inf sample.
An unobserved drop now increments the new
pg_exporter_collector_scrape_aborted_total{collector} counter and sets
last_scrape_success to 0. The duration is still observed, now documented as
time-until-abort, so a stalled collector still shows up in _sum / _count.

A slow or hung OS read could pile tasks onto the blocking pool. A started
spawn_blocking task cannot be cancelled, so once the gate began reopening on
timeout (#34), a sample that outlives its scrape - an over-timeout PSS walk,
or ssl_cert_file on a hung mount - accumulated one queued blocking-pool task
per scrape, unbounded. Sampling is now coalesced: the process-group collector
takes its baseline lock with try_lock and a scrape that finds a sample
already running skips its own (the in-flight one publishes newer data
anyway), and the TLS certificate read holds a one-slot lock so a hung read
leaks exactly one blocking thread instead of one per scrape. A side effect
worth having: with a walk slower than the scrape timeout, scrapes that
overlap an in-flight walk now succeed serving the last published
process-group gauges instead of every scrape timing out.

Measured with a 2GB shared_buffers container under pgbench load (PSS walk
~1.2-2.5s) and --scrape.timeout-ms 500, 50 scrapes: before, 50/50 returned
504 while the process thread count climbed 34 -> 69 (~1/s, unbounded) and 17
aborts were recorded as phantom successes; after, 45/50 returned 200, th... (continued)

Coverage Stats

272 of 296 new or added lines in 6 files covered. (91.89%)

6 existing lines in 2 files now uncovered.

19705 of 21946 relevant lines covered (89.79%)

479.69 hits per line