NVIDIA / nodewright / 38603377493

84%

Build Type

push

github

Committed by web-flow

Commit Message

feat(operator): key the JobReconciler workqueue by node (#789)

* fix(operator): stop retrying Jobs with an unreadable package annotation

A package annotation that fails to unmarshal is not transient, but
JobReconciler returned the error, so the Job was requeued forever and
backed up the controller. Treat it like a missing annotation: log it
and let a terminal Job be marked processed so it TTLs out.

Part of #382.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Steven Stolarski <sstolarski@nvidia.com>

* feat(operator): key the JobReconciler workqueue by node

JobReconciler ran at the default MaxConcurrentReconciles of 1, so every
package-stage Job in the cluster reconciled serially and one failing Job
backed up all nodes. Raising concurrency alone is unsafe: patchNodeState
locks on the whole Node's resourceVersion, so any two concurrent patches
to one Node conflict.

Enqueue the Job's node, with the Job's namespace, instead of the Job. The
node name comes from the pod template's nodeName, not the hash-truncated
label. The workqueue never hands one key to two workers, so a node's Jobs
reconcile serially while different nodes run in parallel, and
MaxConcurrentReconciles is raised to 4.

A pass reconciles every unprocessed Job on the node. A Job whose
reconcile errors cannot hold back the others: its error is returned after
the rest were attempted, and backs off only that node. There is no
requeue of its own, because each outcome ends in a write to the Job whose
event re-enqueues the node. The one timed requeue is the smallest
FailureTarget grace window among the node's Jobs.

The list is scoped to the request's namespace, jobNodeLabel is shared by
the writer and every node-scoped list, and comments that described
per-Job keying are updated.

Part of #382.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Steven Stolarski <sstolarski@nvidia.com>

* docs(operator): describe the node-keyed JobReconciler q... (continued)

Coverage Stats

56 of 60 new or added lines in 3 files covered. (93.33%)

10 existing lines in 2 files now uncovered.

10004 of 12003 relevant lines covered (83.35%)

7.61 hits per line