topfreegames / pitaya / 27216433365

61%

master: 69%

Build Type

push

github

Committed by web-flow

Commit Message

feat(metrics): instrument handler/RPC channel buffers and worker pools (#500)

Several buffered channels and goroutine pools that sit on the hot path
had no instrumentation, so there was no way to tell from metrics when a
server was hitting its buffer (BUFFER_*) or concurrency (CONCURRENCY_*)
limits — the channels would just fill and apply backpressure silently.

Add capacity/utilization metrics for the previously-unmonitored ones:

Channel capacity (reuses channel_capacity_histogram, new `channel` tags):
  - handler_chlocalprocess   (buffer.handler.localprocess)
  - handler_chremoteprocess  (buffer.handler.remoteprocess)
  - rpc_server_userkickchan  (shares the nats messages buffer)

Worker-pool utilization (new gauges pitaya_worker_pool_busy_workers and
pitaya_worker_pool_total_workers, tagged by `pool`):
  - handler_dispatch    (concurrency.handler.dispatch)
  - rpc_server_remote   (cluster.rpc.server.nats.services)

Implementation:
  - Handler dispatch channels and pool are sampled periodically from
    App.periodicMetrics (no extra work on the per-message path).
  - userKickCh and the remote pool are reported from the existing nats
    reportMetrics, alongside the channels already covered there.
  - Add metrics.ReportChannelCapacity / ReportWorkerPoolUsage helpers and
    refactor nats reportMetrics to use them; each call builds its own
    label map so prometheus's label mutation can't leak across reporters.

Busy-worker counts are tracked with atomic int32 counters (two adds per
message, dwarfed by the actual handler/RPC work). Purely additive — no
existing metric, behaviour, or config is changed; works on both the
prometheus and statsd reporters.

Coverage Stats

26 of 84 new or added lines in 5 files covered. (30.95%)

5 existing lines in 3 files now uncovered.

5191 of 8445 relevant lines covered (61.47%)

0.68 hits per line