stacklok / toolhive / 32264738558

70%

Build Type

push

github

Committed by web-flow

Commit Message

Reuse tool embeddings across sessions (#5996)

* Reuse tool embeddings across sessions

The optimizer re-embeds the whole tool set on every client session, so
tools/list blocks on a full index build each time a client connects. At 140
aggregated tools against a CPU embedding backend that is 16-19s per connect,
and under concurrency the redundant rebuilds queue until sessions fail.

THV-0022 describes the store as a regenerable cache whose cold-start cost
falls on the first session after a pod restart. This restores that: an
embedding is reused when the tool's embedded text and the embedding backend
are both unchanged, keyed on a hash of the text plus the backend identity.

Because vectors now outlive a single build, two things follow. Stored vectors
whose width differs from the current backend's are skipped in search rather
than compared, since cosine distance indexes both slices positionally. And a
fixed probe string is re-embedded on each build and compared with the stored
one, because neither the content hash nor the vector width can observe a
same-width model swap behind an unchanged service URL.

Fixes #5847

Signed-off-by: TANTIOPE <antiope.tristan.pro@gmail.com>

* Derive the cache identity from the backend model id, dropping the canary

The embedding cache key now folds in the model id read from the backend
on every build: TEI reports it from /info, the OpenAI client knows it
from configuration. A model swap changes the keys, so stale vectors
stop being found instead of needing to be detected and discarded —
which makes the canary probe, its table, its ordering lock and its
distance threshold unnecessary. A failed id read falls back to the last
id seen, keeping keys stable across transient failures.

The identity is re-read after each embedding batch; a build whose batch
spanned a swap is discarded and re-run under the new identity rather
than committing vectors under keys naming the wrong model.

The dimension guard moves into CosineSimilarit... (continued)

Coverage Stats

205 of 239 new or added lines in 5 files covered. (85.77%)

89 existing lines in 7 files now uncovered.

85996 of 122027 relevant lines covered (70.47%)

100.99 hits per line