stacklok / toolhive-cloud-ui / 25161473077

78%

push

github

fix: make assistant E2E tests model-agnostic and upgrade to qwen3 (#498)

* fix: make assistant E2E tests model-agnostic and upgrade to qwen3

The assistant E2E tests were flaky because they asserted on specific
LLM output patterns (regex for sequential numbers, greeting with
username). This breaks with models that have thinking mode (qwen3)
or different output formatting.

Changes:
- Replace two flaky tests with a single deterministic arithmetic
  test ("What is 1 + 1?" → check response contains "2")
- Increase beforeAll warmup timeout from 30s to 120s — qwen3 with
  thinking mode takes ~40s for first inference on CI runners
- Upgrade E2E model from qwen2.5:1.5b to qwen3:1.7b in workflow
  and all source file defaults

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use testInfo.setTimeout and scope assertion to sidebar

- Use testInfo.setTimeout in beforeAll with biome-ignore for the
  required empty destructuring (Playwright 1.58 does not support
  the { timeout } option on beforeAll)
- Scope the /\b2\b/ assertion to the assistant sidebar
  (data-side="right") to avoid false positives from unrelated page
  content like pagination or counts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

246 of 325 branches covered (75.69%)

469 of 597 relevant lines covered (78.56%)

13.31 hits per line