topfreegames / pitaya / 27141435631

61%

master: 69%

Build Type

Pull #500

github

Committed by rsafonseca

Commit Message

feat(metrics): instrument handler/RPC channel buffers and worker pools

Several buffered channels and goroutine pools that sit on the hot path
had no instrumentation, so there was no way to tell from metrics when a
server was hitting its buffer (BUFFER_*) or concurrency (CONCURRENCY_*)
limits — the channels would just fill and apply backpressure silently.

Add capacity/utilization metrics for the previously-unmonitored ones:

Channel capacity (reuses channel_capacity_histogram, new `channel` tags):
  - handler_chlocalprocess   (buffer.handler.localprocess)
  - handler_chremoteprocess  (buffer.handler.remoteprocess)
  - rpc_server_userkickchan  (shares the nats messages buffer)

Worker-pool utilization (new gauges pitaya_worker_pool_busy_workers and
pitaya_worker_pool_total_workers, tagged by `pool`):
  - handler_dispatch    (concurrency.handler.dispatch)
  - rpc_server_remote   (cluster.rpc.server.nats.services)

Implementation:
  - Handler dispatch channels and pool are sampled periodically from
    App.periodicMetrics (no extra work on the per-message path).
  - userKickCh and the remote pool are reported from the existing nats
    reportMetrics, alongside the channels already covered there.
  - Add metrics.ReportChannelCapacity / ReportWorkerPoolUsage helpers and
    refactor nats reportMetrics to use them; each call builds its own
    label map so prometheus's label mutation can't leak across reporters.

Busy-worker counts are tracked with atomic int32 counters (two adds per
message, dwarfed by the actual handler/RPC work). Purely additive — no
existing metric, behaviour, or config is changed; works on both the
prometheus and statsd reporters.

Pull Request Pull Request #500: feat(metrics): instrument handler/RPC channel buffers and worker pools

Coverage Stats

26 of 84 new or added lines in 5 files covered. (30.95%)

21 existing lines in 4 files now uncovered.

5173 of 8445 relevant lines covered (61.26%)

0.68 hits per line