stacklok / toolhive / 32205375602

70%

Build Type

push

github

Committed by web-flow

Commit Message

Serve Prometheus metrics on a separate diagnostics listener (#6296)

* Serve Prometheus metrics on a separate listener

Go's ServeMux resolves the most specific registered pattern first, so the
explicitly registered /metrics always outranked the "/" catch-all that
carries the proxy middleware chain. The endpoint was therefore reachable
without authentication, body limits, rate limiting, or audit even on a
fully OIDC-configured deployment. In Kubernetes it was also internet-
reachable: the operator binds the proxy to 0.0.0.0 and the Service maps
the proxy port.

Bind metrics to a dedicated diagnostics listener instead. The runner no
longer hands the Prometheus handler to the transport, which is what keeps
the proxies from mounting /metrics on the application mux at all; they now
return 404 there, mirroring how /health already guards itself. /health
stays on the application listener because Kubernetes probes target it and
it exposes no build information.

The listener defaults to port 9464, the OpenTelemetry specification's
Prometheus exporter default, and falls back to an available port when that
one is taken. This matches the pattern the ToolHive operator already uses
for its own metrics endpoint (--metrics-bind-address), as do etcd
(--listen-metrics-urls) and controller-runtime.

Part of #6271

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Harden diagnostics listener lifecycle and timeouts

Release the listener on every exit from Runner.Run. Cleanup runs only via
stopMCPServer, which the early error returns skip, as does the graceful
container-exit branch that returns nil after the transport has stopped.
Leaking there stranded a goroutine and a bound port; under the restart
loop in workloads.Manager each attempt would strand another and push the
next onto a different port, silently breaking the stable scrape target
this change exists to provide.

Set ReadTimeout, WriteTimeout, and MaxHeaderBytes to match the proxy and
vMCP lis... (continued)

Coverage Stats

146 of 179 new or added lines in 9 files covered. (81.56%)

79 existing lines in 5 files now uncovered.

85595 of 121529 relevant lines covered (70.43%)

99.22 hits per line