stacklok / toolhive / 21288691742

push

github

fix(#3411) - waitForStatefulSet waits for the latest generation (#3413)

Fixes a race condition in waitForStatefulSetReady that causes the proxy to start before the new MCP pod is ready during rolling updates, leading to DNS lookup failures and restart loops.

Closes #3411

Problem
During a StatefulSet rolling update, waitForStatefulSetReady only checked ReadyReplicas == Replicas. This could return "ready" when:

The OLD pod was still ready (ReadyReplicas = 1)
The controller hadn't processed our spec yet (ObservedGeneration < desiredGeneration)
No pods were updated to the new spec (UpdatedReplicas = 0)
The proxy would start, attempt to connect to the headless service, but the backing pod was being rolled. DNS lookups failed with no such host, the health monitor detected the failure, and both pods entered a restart loop.

Solution
Update isStatefulSetReady to check all three conditions before considering the StatefulSet ready:

ObservedGeneration >= desiredGeneration (controller processed our spec)
UpdatedReplicas == Replicas (all pods on new spec)
ReadyReplicas == Replicas (all pods ready)
Changes
The commit history incrementally introduces changes for easy reading:

Extract isStatefulSetReady as a private, unit-testable function. Add unit tests demonstrating the bug.
Add desiredGeneration parameter to waitForStatefulSetReady to track the generation from the Apply call.
Fix the bug by waiting until the statefulset is on the desiredGeneration.

8 of 13 new or added lines in 1 file covered. (61.54%)

36991 of 61817 relevant lines covered (59.84%)

79.35 hits per line