NVIDIA / nodewright / 38653367903

84%

Build Type

push

github

Committed by web-flow

Commit Message

fix(operator): leave kubelet admission rejections out of a package stage's Restarts (#787)

* fix(operator): leave admission rejections out of a package stage's Restarts

A package stage's Restarts came straight from its Job's status.failed,
which also counts the pods the kubelet rejected at admission before the
package ran. A node refusing replacements on its way back from a reboot
then showed four failed attempts, in the CLI and in
nodewright_package_restarts_count, for a package that never executed.

Every Job-side Restarts write for a package stage (recordJobRestarts,
recordJobErroring, recordJobCompletion) now goes through one helper,
packageJobRestarts: status.failed less the Job's child pods rejected at
admission, never below 0. A rejected pod is one podRejectedAtAdmission
accepts (Failed, no container statuses, no DisruptionTarget, not
podFailedGenuinely), and it is subtracted only once the Job has counted
it, that is when status.uncountedTerminatedPods.failed no longer lists
it. Disruption casualties, ContainerStatusUnknown pods and pods killed by
their own deadline are still counted. The rejections still spend the
Job's retry budget.

The pods are read from another cache than the Job, so an in-flight
count can be briefly off. Every Job status write reruns it, and a
terminal Job has no uncounted pods. If the pod list fails, the write
falls back to status.failed so a display count never holds up the state
transition it rides on. Interrupt Jobs keep their current values.

Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>

* docs(observability): say a package stage's restarts leave out admission rejections

A package stage's Restarts no longer counts the pods the kubelet
rejected at admission, but the metric reference in metrics.md, the
nodewright_package_restarts_count HELP string and the Current Package
Status panel description still said it counted every failed pod of the
stage's Job.

All three now say a package stage counts its Job's ... (continued)

Coverage Stats

37 of 39 new or added lines in 2 files covered. (94.87%)

33 existing lines in 4 files now uncovered.

10039 of 12038 relevant lines covered (83.39%)

7.62 hits per line