cweill / gotests / 18735731293 / 1
94%
master: 94%

LAST BUILD BRANCH: develop
DEFAULT BRANCH: master
Ran
Files 14
Run time 0s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 94.333% (-1.9%) from 96.272%
18735731293.1

Pull #194

github

cweill 
feat: add retry logic to E2E tests for non-deterministic LLM output

**Problem:**
Small LLM models like qwen2.5-coder:0.5b are not perfectly deterministic even
with temperature=0. In CI, the model sometimes generates slightly different
test case names or argument values compared to the golden files.

**Solution:**
- Added retry logic (up to 3 attempts) to E2E tests
- Extract validation logic into `compareTestCases()` helper function
- Tests now retry generation if output doesn't match golden file
- Only fail if all 3 attempts produce mismatched output
- Log which attempt succeeded for debugging

**Benefits:**
- E2E tests now handle LLM variance in CI environments
- Still validates that AI generation works end-to-end
- Provides better debugging info when tests fail (errors from last attempt)
- Maintains strict validation - just adds tolerance for variance

**Example Log:**
```
✓ Generated 3 test cases for ParseKeyValue (attempt 1/3)
✓ Matched golden file on attempt 2/3  # if retry needed
```

**Testing:**
- All 11 E2E tests pass locally (all matched on attempt 1/3)
- Retry logic verified with refactored validation function
- No changes to validation strictness - same checks applied each attempt

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Pull Request #194: feat: AI-powered test case generation

1548 of 1641 relevant lines covered (94.33%)

1325.12 hits per line

Source Files on job 18735731293.1