opendatahub-io / trainer / 24092326438

55%

main: 55%

push

github

Sanitize NaN/Infinity in metrics JSON before parsing

Python's json.dumps() allows NaN and Infinity by default, but these
are not valid JSON per the spec. When training metrics contain these
values (e.g., grad_norm when loss=0, eval_loss with bf16 on ROCm),
Go's json.Unmarshal fails, causing the operator to never capture
training progress.

Add sanitizeJSON() to replace NaN/Infinity with null before parsing
in both PollTrainingProgress and CaptureMetricsFromTerminationMessage.
The regex patterns are anchored to trailing JSON delimiters to avoid
corrupting string values.

Ref: RHOAIENG-56898

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

17 of 17 new or added lines in 1 file covered. (100.0%)

1685 of 3052 relevant lines covered (55.21%)

0.65 hits per line