kubeflow / trainer / 32153247293

68%

push

github

feat(controller): register TrainJobStatus server with controller manager healthz and readyz probes (#3338)

The TrainJobStatus server runs inside the controller manager process but was
not wired into its /healthz and /readyz endpoints. If the server died mid-job
the controller pod stayed Running with no signal to the kubelet, and training
nodes had no way to tell whether the server was ready to accept their first
status update while TLS and the OIDC provider were still initializing.

RegisterProbes adds a single checker to both endpoints. The checker dials the
server's TLS listener, so it only reports healthy once the server is genuinely
accepting connections, mirroring how controller-runtime probes the webhook
server. It is called from main.go before mgr.Start(), because controller-runtime
rejects check registrations once the manager is running.

Also close the status server's shutdown channel unconditionally. Start() waits
on that channel after ListenAndServeTLS returns, and nothing ever closed it, so
a graceful shutdown blocked forever and the manager never finished stopping.

Signed-off-by: krishdef7 <gargkrish06@gmail.com>

20 of 30 new or added lines in 2 files covered. (66.67%)

2602 of 3772 relevant lines covered (68.98%)

0.8 hits per line