stacklok / toolhive / 22772672652

64%

Build Type

push

github

Committed by web-flow

Commit Message

Auto-chunk TEI embedding requests to respect server batch size limit (#4029)

TEI (Text Embeddings Inference) does not automatically batch embedding
requests — it rejects requests that exceed max_client_batch_size with a
422 error. This wasn't caught earlier because the Optimizer was indexing
fewer than 32 tools (the default limit). Once we exceeded that threshold,
upserts started failing.

Query the TEI /info endpoint at client creation to discover
max_client_batch_size, then split EmbedBatch requests into chunks
that fit within the limit. Falls back to a default of 32 when
/info is unavailable.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Run Details

47 of 47 new or added lines in 1 file covered. (100.0%)

10 existing lines in 2 files now uncovered.

47291 of 73977 relevant lines covered (63.93%)

75.08 hits per line