medplum / medplum / 30569767210

92%

push

github

feat(server): add queue activeCount, per-queue in-flight and completed job metrics, and async batch entry throughput (#10005)

* feat(server): add queue activeCount, per-host in-flight jobs, and async batch rates

Adds three kinds of queue observability to the OpenTelemetry heartbeat:

- `medplum.<queue>.activeCount` gauge, alongside the existing waiting and
  delayed counts.
- `medplum.<queue>.inFlightJobs` UpDownCounter, maintained per-process by every
  worker via a new `trackInFlightJobs` processor wrapper. This wraps the
  processor rather than listening for the worker's active/completed/failed
  events because those do not pair up: BullMQ suppresses `failed` when a job is
  moved to delayed, and suppresses both terminal events once the connection is
  closing, either of which would leave the counter permanently incremented.
- `medplum.batch.completedPerSecond` and `medplum.batch.entriesPerSecond`,
  derived by a new `setRateGauge` helper from running totals the batch worker
  keeps. The entry rate covers the re-entrant path only; the deprecated legacy
  path hands the whole bundle to the router at once.

Per-queue metric names now go through `getQueueMetricName`, keeping the queue in
the metric name rather than an attribute to match how these have always been
reported. `QueueMetric` documents why `activeCount` and `inFlightJobs` measure
different things and are not expected to agree.

Also adds `getUpDownCounter`/`addToUpDownCounter` helpers, and documents the
UpDownCounter instrument in the otel README.

Signed-off-by: Derrick Farris <derrick@medplum.com>

* refactor(server): count async batch throughput instead of computing a rate

Replaces the `medplum.batch.completedPerSecond` and
`medplum.batch.entriesPerSecond` gauges, and the `setRateGauge` helper that
backed them, with two plain monotonic counters:

- `medplum.batch.job.completed.count`
- `medplum.batch.entry.processed.count`

Deriving the rate is now the observability backend's job, whi... (continued)

22743 of 25823 branches covered (88.07%)

Branch coverage included in aggregate %.

27 of 32 new or added lines in 9 files covered. (84.38%)

40333 of 42717 relevant lines covered (94.42%)

12259.52 hits per line