NVIDIA / nodewright / 35928098961
83%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 59
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 82.426% (-0.1%) from 82.541%
35928098961

push

github

web-flow 
fix(agent): let the running step finish on SIGTERM so gracefulShutdown holds (#673)

The Go agent SIGKILLed the in-flight step the instant its pod received
SIGTERM. cmd/agent/main.go cancels its context on SIGTERM, and
internal/command/process.go bound every child to that context through
exec.CommandContext with a Cancel that killed the whole process group. Any
pod teardown mid-step -- drain, eviction, kubectl delete pod, operator pause
-- killed the script at an arbitrary point in a host mutation, and the
package's gracefulShutdown, documented as the time a script has to finish,
bought it nothing. The Python agent lets the step finish and only stops
before the next one.

Run the child with exec.Command instead, so cancellation no longer
terminates it. The existing ctx.Err() checks -- before a command starts, and
between steps and between interrupt operations -- still refuse to start
anything once SIGTERM has arrived, so the semantics become "finish what is
running, then stop", matching Python and the documentation. A hung step is
now bounded by the kubelet at the end of the grace period, which is what
gracefulShutdown is for. NodeRestart is preserved, and a race is removed
along the way: the old Cancel could SIGKILL the reboot child before
systemd's SIGTERM reached it, misreporting a completed reboot as a failure.

The post-run ctx.Err() attribution is dropped too. The child is never
terminated on cancellation, so a failure there is its own and is reported
with its real exit status.

One deliberate difference from Python remains: Python runs every remaining
interrupt operation after SIGTERM, since do_interrupt never checks for it;
the Go agent finishes the current operation and refuses the next. Both let
the running command finish. Stopping between operations is the safer of the
two, and completion markers are written per operation so a retry resumes at
the right place.

Three tests asserted the old behaviour and are rewritten to the corrected
expect... (continued)

9362 of 11358 relevant lines covered (82.43%)

7.63 hits per line

Coverage Regressions

Lines Coverage ∆ File
8
77.49
-2.28% operator/internal/cli/utils/utils.go
5
78.34
-3.18% operator/cmd/cli/app/lifecycle.go
Jobs
ID Job ID Ran Files Coverage
1 35928098961.1 59
82.43
GitHub Action Run
Source Files on build 35928098961