zwave-js / zwave-js-ui / 30525098149

12%

push

github

feat(bot): judge docs answers with an agentic workflow (#4765)

## Description

Last of three PRs migrating off GitHub Models (based on #4764, porting
zwave-js/zwave-js#8987). The docs answer bot's gpt-4o judge moves to a
gh-aw workflow on the Copilot engine, split so only the judgment is
agentic:

- **Pre-activation** keeps all deterministic gates (excluded users,
categories, existing answers, suppression list) plus the index
restore/health machinery (`restore-bot-index`, `report-index-status`),
retrieves the doc excerpts, posts related-posts-only comments directly,
and only gates the engine on when doc excerpts need judging
- **The judge** reads a projected `judge-input.json` (question + excerpt
text, no embedding vectors) and reports
`confidence`/`answer`/`related_excerpts` through a typed safe-output
job, capped at 5 turns with no GitHub toolset and no tool egress. Its
prompt is the former `judgeAnswer` system prompt, including the
untrusted-input rule
- **The safe-output job** validates the verdict (length-capped, tolerant
id parsing) and composes/posts the comment with the same confidence
thresholds, answer sanitizer, link dedup and feedback metadata (now
including the run id) as before, and re-checks `alreadyAnswered` to
avoid duplicate answers from overlapping runs

This removes the last GitHub Models consumer: `modelsApi.cjs` and the
eval's `--answers` mode are gone, and no workflow requests `models:
read` anymore.

## Validation

- Mocked end-to-end dry run against real rebuilt indexes: on-topic
question → handoff with the right excerpts, engine gated on; off-topic →
everything gated off; judge verdict → correctly composed comment with
doc links
- Unit tests cover the render thresholds, all four comment shapes, the
answer gates, and a prepare→post round trip pinning the handoff
contract; `gh aw compile` clean, all 156 bot-script tests pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fabl... (continued)

493 of 4531 branches covered (10.88%)

Branch coverage included in aggregate %.

938 of 7002 relevant lines covered (13.4%)

2.08 hits per line