kubernetes-sigs / inference-perf / 36556385492

81%

Build Type

push

github

Committed by web-flow

Commit Message

feat(datagen): support wire-captured traces and lazy local trace loading (#820)

## Summary

  This PR extends OTel trace replay with:

  - Support for captured API wire traces:
      - OpenAI Chat Completions
      - OpenAI Responses API
      - Anthropic Messages API

  - Support for line-delimited OTel .jsonl files.
- Lazy loading of local traces to avoid constructing one large in-memory
Arrow dataset.
- Preservation of existing local OTel .json behavior, including session
identity.
- Correct expansion of parallel tool results into individual role: tool
messages.

• ## Motivation

  This PR has two goals:

  1. **Replay wire-capture traces**

Support replaying captured API requests and responses directly, without
requiring a separate conversion step. One source of these traces is
llm-d live benchmarking with Exgentic, which produces wire captures that
can then be reused as replay workloads.

  2. **Improve local-trace loading at scale**

Local traces were previously combined into a single Hugging Face
`Dataset`. For large corpora, this increased startup memory usage and
could exceed Arrow’s 32-bit offset limits.

Local traces are now indexed using lightweight record locators and
loaded individually when their sessions are built. This avoids
materializing the complete local corpus in memory while preserving
existing OTel JSON replay behavior.

---------

Co-authored-by: Alon Halfon <alonhal@il.ibm.com>

Coverage Stats

672 of 829 new or added lines in 4 files covered. (81.06%)

3 existing lines in 1 file now uncovered.

11375 of 13995 relevant lines covered (81.28%)

0.81 hits per line