topfreegames / pitaya / 27143603019

59%

master: 69%

Build Type

Pull #501

github

Committed by rsafonseca

Commit Message

feat(metrics): instrument handler/RPC channel buffers and worker pools

Several buffered channels and goroutine pools that sit on the hot path
had no instrumentation, so there was no way to tell from metrics when a
server was hitting its buffer (BUFFER_*) or concurrency (CONCURRENCY_*)
limits — the channels would just fill and apply backpressure silently.

Add capacity/utilization metrics for the previously-unmonitored ones:

Channel capacity (reuses the channel_capacity histogram, new `channel` tags):
  - handler_chlocalprocess   (buffer.handler.localprocess)
  - handler_chremoteprocess  (buffer.handler.remoteprocess)
  - rpc_server_userkickchan  (shares the nats messages buffer)

Worker-pool utilization (new gauges pitaya_worker_pool_busy_workers and
pitaya_worker_pool_total_workers, tagged by `pool`):
  - handler_dispatch    (concurrency.handler.dispatch)
  - rpc_server_remote   (cluster.rpc.server.nats.services)

Implementation:
  - Handler dispatch channels and pool are sampled periodically from
    App.periodicMetrics (no extra work on the per-message path).
  - userKickCh and the remote pool are reported from the existing nats
    reportMetrics, alongside the channels already covered there.
  - Add metrics.ReportChannelCapacity / ReportWorkerPoolUsage helpers and
    refactor nats reportMetrics to use them; each call builds its own
    label map so prometheus's label mutation can't leak across reporters.

Busy-worker counts are tracked with atomic int32 counters (two adds per
message, dwarfed by the actual handler/RPC work). Purely additive — no
existing metric, behaviour, or config is changed; works on both the
prometheus and statsd reporters.

Pull Request Pull Request #501: feat(metrics): instrument handler/RPC channel buffers and worker pools

Coverage Stats

26 of 81 new or added lines in 5 files covered. (32.1%)

5 existing lines in 3 files now uncovered.

5105 of 8720 relevant lines covered (58.54%)

0.65 hits per line