stacklok / toolhive / 29824312474

69%

Build Type

push

github

Committed by web-flow

Commit Message

Warm ingress Squid peer connections with standby=2 (#5887)

Non-stdio workloads are fronted by a Squid ingress reverse proxy. Its
cache_peer opened upstream connections lazily, so on a cold start the
first GET /mcp SSE stream paid inline DNS + TCP connect latency while a
later tools/call POST reused a warmed path. The POST then reached the
server and ran its handler before the standalone GET SSE stream was
established, leaving a server-initiated request (e.g. sampling) with no
stream to ride until the client's 60s timeout.

Add standby=2 to the ingress cache_peer so Squid maintains warm idle
upstream connections; the first GET is forwarded without inline connect
latency, removing the GET-vs-POST reorder. Config-only, no ACL change.

Verified on the diagnostics branch: a cold-start conformance comparison
went from ~1/3 (squid-only) and ~2/3 (full stack) sampling failures to
0/8 on every tier with this change.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Coverage Stats

6 of 6 new or added lines in 1 file covered. (100.0%)

7 existing lines in 2 files now uncovered.

75704 of 109977 relevant lines covered (68.84%)

72.89 hits per line