kubernetes-sigs / inference-perf / 36779672373

81%

push

github

feat(sglang): support the embeddings API (#850)

Fixes #849

Lets `server.type: sglang` run `api.type: embeddings` benchmarks.

## What changed

- `SGlangModelServerClient.get_supported_apis()` now includes
`APIType.Embeddings`, like the vLLM client.
- No request or response changes: SGLang serves `/v1/embeddings` in the
OpenAI format when started with `--is-embedding`.
- `examples/embeddings/README.md` lists `sglang` and shows how to start
it.

## Testing

- New `test_embeddings_request` sends an embeddings request through the
SGLang client and parses a response captured from SGLang 0.5.20. It
fails without the fix.
- `pytest tests/required`, `ruff check`, `ruff format --check` and `mypy
--strict ./inference_perf ./tests` pass.
- Real SGLang 0.5.20 serving `BAAI/bge-small-en-v1.5` on an RTX 4060
laptop GPU, using `examples/embeddings/config.yml` (400 requests, 8 in
flight):

  | batch_size | requests/s | inputs/s | p50 latency | p99 latency |
  |-----------:|-----------:|---------:|------------:|------------:|
  | 1  | 104.4 | 104 | 66 ms   | 101 ms   |
  | 16 | 27.8  | 445 | 250 ms  | 908 ms   |
  | 64 | 7.1   | 453 | 1031 ms | 1289 ms  |

  All 1,200 requests succeeded, and TTFT/TPOT/ITL were `null`.

## Note

With this model, SGLang's default FlashInfer attention backend fails on
the first request (`Invalid configuration`). `--attention-backend
triton` fixes it, and the README says so.

1 of 1 new or added line in 1 file covered. (100.0%)

11375 of 13995 relevant lines covered (81.28%)

0.81 hits per line