kubernetes-sigs / inference-perf / 38617044466

86%

push

github

fix: send chat requests for text-only shared_prefix with api.type chat (#893)

Fixes #892

With `api.type: chat`, text-only `shared_prefix` data went to
`/v1/completions`. #450 listed Chat as supported but built chat requests
only for multimodal configs.

Text-only chat requests send each prompt as one user message, the same
text a completion request sends. Multimodal configs keep their own path.
#892 proposed building them with `prefix_text` like the multimodal path.
That rebuilds the prompt from the prefix and the question, sends a list
of content parts and logs multimodal progress for text-only runs. One
string is simpler and is exactly the completion prompt.
`tests/optional/text/cases/chat` uses this combination and expects chat
requests. #287 had the same goal and was closed for inactivity.

Breaking change: `enable_multi_turn_chat` with `api.type: chat` fails at
startup with a config error. It used to run and send completion
requests, since multi-turn sessions build only completion requests. Such
configs should set `api.type: completion`.

Testing:
- New tests: a chat config builds `/v1/chat/completions` requests whose
message is the completion prompt, a completion config still builds
completions and multi-turn with chat raises. The chat and multi-turn
tests fail before the fix.
- Against a stub server, the config from #892 sent all 12 requests to
`/v1/chat/completions`. With multi-turn added, the run exits with the
config error.
- `pytest tests` passes (2310 passed). So do `ruff check`, `ruff format
--check` and `mypy --strict`.

---------

Signed-off-by: Vu Dinh <vudinh@outlook.com>

4 of 4 new or added lines in 1 file covered. (100.0%)

12088 of 14502 relevant lines covered (83.35%)

0.83 hits per line