christophevg / yoker-test

98%

push

github

feat(usage): capture backend credit usage and rank models by score-per-cost (P2.5.10)

Capture Ollama credit usage during evaluations via injected backend:
fetch_usage snapshots (session + weekly request deltas), with
quantized-zero persistence, negative-delta dropping, and usage-API
unavailability notes in test results.

Rank models by score-per-cost: a composite score (quality primary,
usage-weighted secondary) with weekly usage deltas in the quality
ranking and composite-proximity tie flagging in the console report.

- usage.py: extract/fetch normalized usage metrics from backend payloads
- runner.py: backend injection, snapshot cadence, circuit breaker,
  normalized usage deltas on TestResult
- schema.py: usage fields (requests/cost/usage_note) with round-trip
  compatibility for pre-existing results
- report.py: composite ranking, weekly delta column, proximity ties
- README.md: document usage capture and score-per-cost ranking
- tests: 415 tests, make check green

🤖 Implemented together with Yoker.

277 of 290 branches covered (95.52%)

Branch coverage included in aggregate %.

941 of 957 relevant lines covered (98.33%)

0.98 hits per line