NVIDIA / k8s-driver-manager / 34391032241

11%

push

github

retry the cordon and uncordon of a node on transient API errors (#210)

k8s-driver-manager cordons the node, performs its driver work, then
uncordons it. Both calls gave up on the first API error, so a transient
API-server timeout, 5xx, or connection refused could abort an upgrade or
leave the node cordoned. We hit this as clusters come up for the first
time, when the API server is not yet fully ready; it is not an issue
during steady state or upgrades.

Wrap both calls in a bounded exponential backoff (5 attempts, 1s base,
factor 2, 20% jitter). The retry policy lives at the call site, in
driver-manager's cordonNode and uncordonNode, so the kube client stays a
thin wrapper over the Kubernetes API and carries no retry concern.
Because the client reads the Node inside CordonNode and UncordonNode, a
retried call always patches a freshly read object rather than a stale
one. How each call site treats an exhausted budget is unchanged.

The retry loop is context aware: a cancelled context ends it promptly
instead of sleeping out the remaining backoff, and is joined with the
last error via errors.Join so both causes stay inspectable rather than
being conflated, which is what wait.Interrupted would do. Failed
attempts are logged with their attempt number and only described as
retrying while attempts remain, so the attempt that exhausts the budget
is logged as a final failure instead.

There is no new env var or flag: the retry limit is a constant.

The retry helper is unit tested directly. The client's per-attempt
contract is covered by driving the real typed clientset against an
httptest stand-in API server with independent failure budgets for reads
and writes, so a failed GET and a failed PATCH are both exercised.
(client-go's fake clientset is not vendored, and vendoring it would add
a large unrelated diff.)

Signed-off-by: Cody Eilar <cody@hyperbolic.xyz>

35 of 48 new or added lines in 2 files covered. (72.92%)

187 of 1689 relevant lines covered (11.07%)

0.13 hits per line