opendatahub-io / trainer / 24070722781

55%

main: 54%

Build Type

Pull #139

github

Committed by sutaakar

Commit Message

Sanitize NaN/Infinity in metrics JSON before parsing

Python's json.dumps() allows NaN and Infinity by default, but these
are not valid JSON per the spec. When training metrics contain these
values (e.g., grad_norm when loss=0, eval_loss with bf16 on ROCm),
Go's json.Unmarshal fails, causing the operator to never capture
training progress.

Add sanitizeJSON() to replace NaN/Infinity with null before parsing
in both PollTrainingProgress and CaptureMetricsFromTerminationMessage.
The regex patterns are anchored to trailing JSON delimiters to avoid
corrupting string values.

Ref: RHOAIENG-56898

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Pull Request Pull Request #139: Sanitize NaN/Infinity in metrics JSON before parsing

Coverage Stats

20 of 20 new or added lines in 1 file covered. (100.0%)

70 existing lines in 1 file now uncovered.

1675 of 3042 relevant lines covered (55.06%)

0.65 hits per line