IntelPython / dpnp / 41065383895

83%

push

github

Use a separate retry timeout for tensor tests in CI (#3132)

The `dpnp.tests.tensor` retry step shared `rerun-tests-timeout` with the
main `dpnp.tests` suite, although the tensor tests normally finish in a
few minutes. The tensor tests sporadically hang (a deadlock in the
OpenCL CPU runtime), and each hang used up the whole shared timeout
before the retry started.

This PR adds a dedicated `rerun-tensor-tests-timeout` (plus
`rerun-tensor-tests-timeout-win` for Windows) and uses it in the tensor
retry steps:

| Workflow | Healthy tensor run | Tensor timeout (before → after) | Main
suite timeout (before → after) |
|---|---|---|---|
| `conda-package.yml`, Linux | up to 5.8 min | 45 → 15 min | 45 min
(unchanged, healthy runs up to ~30 min) |
| `conda-package.yml`, Windows | mostly under 12 min, up to 19.2 min |
45 → 20 min | 45 min (unchanged, healthy runs up to ~34 min) |
| `conda-package-cf.yml` (reusable `conda-build-test.yml`) | as above,
Windows up to 17.6 min | 45 → 15 min (Linux), 20 min (Windows) | 45 min
(unchanged, healthy runs up to ~37 min) |
| `check-onemath.yaml` | 2.4–5.4 min | 20 → 10 min | 20 → 30 min
(healthy runs up to ~19.3 min) |

Windows gets a longer tensor timeout because its healthy runs have a
slow tail: in October, 18 of 688 runs took over 10.5 min and the slowest
took 19.2 min, though none took over 11.7 min since October 16.

In `check-onemath.yaml`, the oneMath develop branch job timeout is also
raised from 60 to 120 min, the same as the oneMath tag job. That job
spends a median of ~31 min building (up to ~37) and ~16 min in the main
suite, so the tensor tests start around minute 51 (up to 58) and even
healthy jobs finish close to the limit (median 54.5 min, p95 59.2 min).
A hung tensor run was therefore cancelled at the job limit before the
retry could fire. Since October 1, 35 develop branch jobs were cancelled
at 60 min: 17 hung at 77–82% in `test_usm_ndarray_linalg.py`, 14 ran out
of time during the tensor tests, and 4 du... (continued)

1604 of 2994 branches covered (53.57%)

Branch coverage included in aggregate %.

29435 of 34567 relevant lines covered (85.15%)

17665.63 hits per line