dcdpr / jp / 24354983231

67%

push

github

fix(llm, anthropic): Inject continue message for non-prefill models (#545)

Some Anthropic models require conversations to end with a user message
and don't support the assistant prefill technique — where an assistant
message as the final event causes the model to continue from that exact
point.

A new `prefill` feature flag has been added to `ModelDetails` to track
which models support this behaviour. When a conversation ends with an
assistant message (e.g. after an interrupt or a retry that flushed
partial content) and the model lacks the `prefill` feature, a synthetic
user "continue" message is injected so the model picks up exactly where
it left off without repeating already-produced content.

All known Anthropic models have been updated with the appropriate
`prefill` flag. The `find_merge_point` helper was also promoted from a
private function in the Anthropic provider to a `pub(crate)` function in
`stream::chain`, removing the duplication that existed between the two
call sites.

Signed-off-by: Jean Mertz <git@jeanmertz.com>

30 of 30 new or added lines in 3 files covered. (100.0%)

21530 of 32982 relevant lines covered (65.28%)

208.13 hits per line