IntelPython / dpnp / 41800409193

84%

push

github

Kill hung tensor tests with pytest-timeout and lower the Windows retry timeout in CI (#3152)

Lower the Windows tensor-tests retry timeout from 20 to 15 minutes and
add a per-test timeout via `pytest-timeout` to the tensor tests in
`conda-package.yml` and `conda-build-test.yml` (used by
`conda-package-cf.yml`).

The `dpnp.tests.tensor` run sporadically hangs at 99%
(https://github.com/IntelPython/dpctl/issues/1954), on both Linux and
Windows. Since October 1, in `conda-package.yml`, 48 of 885 Linux and 25
of 871 Windows tensor steps hit it. The log shows no progress for the
rest of the attempt until the retry timeout kills it, and the next
attempt passes.

| Change | Before | After |
|---|---|---|
| Windows tensor retry timeout | 20 min
(`rerun-tensor-tests-timeout-win`) | 15 min, the same
`rerun-tensor-tests-timeout` as Linux |
| Per-test timeout (Linux and Windows) | none | `--timeout=120
--timeout-method=thread` |

Healthy Windows tensor steps now take 4-6 min, and the slowest took
10-11 min since October 16 (`conda-package-cf.yml`: 4-9 min), so 15 min
leaves room without the 20 min tail kept by #3132.

`--timeout-method=thread` is set explicitly because the hung main thread
waits in native code (`SyclQueue.wait()`), where the default `signal`
method on Linux would never fire. The thread method ends the stuck
pytest-xdist worker: xdist reports the test as failed with its name and
stack, starts a new worker and finishes the run, so a hang fails within
~2 min and the retry reruns the step. This was checked locally with a
test that blocks in native code under `-n 2 --timeout=5
--timeout-method=thread`: `worker 'gw1' crashed while running
'tests/test_hang.py::test_hang'`, `1 failed, 2 passed in 5.58s`.

1792 of 2998 branches covered (59.77%)

Branch coverage included in aggregate %.

29569 of 34170 relevant lines covered (86.53%)

39082.12 hits per line