NVIDIA / nodewright / 37369790357

84%

push

github

fix(operator): throttle PDB-refused eviction attempts per node (#755)

* fix(operator): keep the soonest requeue across NodeWrights in a pass

processSkyhooksPerNode kept the last non-nil result RunSkyhookPackages
returned, so the requeue a reconcile pass asked for depended on the
order NodeWrights were processed in: a later, longer RequeueAfter could
replace an earlier, shorter one.

soonerRequeue now folds each result into the pass result, keeping the
smaller RequeueAfter. Nil results are still ignored and an all-nil pass
still returns nil, so reconcileResult's configSyncRetryInterval and
MaxInterval fallback is unchanged.

Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

* feat(operator): throttle PDB-refused evictions per node, not per pass

Since #625 a PodDisruptionBudget refusal is a wait state rather than a
reconcile error, which also removed the exponential backoff that used to
space out retries: with a budget at zero allowed disruptions the operator
re-attempted the eviction on every 2s pass for as long as it held.

DrainNode now records, per node, when a PDB last refused an eviction
there, and while that refusal is under drainBlockedEvictionInterval (30s)
old it makes no eviction call and returns the blockers the refused
attempt recorded, so the DrainBlocked annotation is kept rather than
cleared. Only the attempt is throttled: the reconcile cadence stays at
2s, IsDrained still runs first on every pass and a drained node drops
its entry, and the drain timeout is checked ahead of the throttle so it
fires on time. Accepted evictions and disableEviction deletes never start
the throttle.

The map is in-memory reconciler state, a deliberate exception to the
no-reconciler-owned-state rule explained at its declaration: losing it
only reverts to an immediate retry. Every pass that builds cluster state
prunes entries for nodes no NodeWright selects. DrainNode reads time from
an injectable k8s.io/utils/clock so specs step a fake clock instead... (continued)

73 of 75 new or added lines in 1 file covered. (97.33%)

9719 of 11716 relevant lines covered (82.95%)

7.64 hits per line