kubernetes-sigs / inference-perf / 38764393792

86%

push

github

fix: use client output_len and preserve Anthropic server_usage (#888)

## Summary
Fixes #885 — 4 provenance bugs found in review on #694.

## Root cause
1. AnthropicMessagesAPIData + SessionAnthropicMessagesAPIData (streaming
+ unary) set output_len to server usage.output_tokens when present.
docs/metrics.md defines output_len as client re-tokenized; chat path
already does that.
2. Unary Anthropic built UnaryResponseMetrics without server_usage, so
count_client_fallbacks counted every unary req as fallback.
3. correct_streamed_response_metrics compared against completion_tokens
only, so Anthropic output_tokens never hit token_count_mismatches.
4. _chunk_content read only delta.text, while live extract_content also
reads content_block_start text.

## Fix
- apis/anthropic_messages.py, datagen/replay/...: always
count_anthropic_output_tokens() for output_len; unary passes
server_usage=usage or None.
- reportgen/base.py: _chunk_content reads start-block text; mismatch
check uses server_reported_tokens(..., SERVER_OUTPUT_TOKEN_KEYS).
- reportgen/per_request/validation.py: _check_token_agreement reads both
completion_tokens and output_tokens (local key loop); warning text
updated.

## User-visible change
Anthropic turns with tool_use used to report the server count as
output_len, tool input included. They now report the text alone, 0 for a
tool-only turn. On agentic replay runs that lowers output throughput and
raises TPOT. use_server_output_tokens: true gets the server count back.

## How tested
- Updated 4 existing anthropic tests to assert client count + preserved
server_usage.
- Added 4 new tests: unary/streaming no-usage fallback, start-block
chunk text, Anthropic-key mismatch detection.
- Added Anthropic twin of
test_client_server_token_divergence_is_a_warning for the per-request
validator.
- pytest tests/required/apis/test_anthropic_messages.py
tests/required/reportgen/...: 57 passed.
- Related suites: 131 passed.
- Full pytest tests: 2108 passed; 6... (continued)

16 of 16 new or added lines in 4 files covered. (100.0%)

12094 of 14508 relevant lines covered (83.36%)

0.83 hits per line