• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

kubernetes-sigs / inference-perf / 42664894421
87%

Build:
DEFAULT BRANCH: main
Ran 09 Nov 2026 05:38PM UTC
Jobs 1
Files 140
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

09 Nov 2026 05:28PM UTC coverage: 86.011% (+0.04%) from 85.976%
42664894421

push

github

web-flow
fix(datagen): schedule conversation replay turns by readiness (#924)

Fixes #906

## Summary

`conversation_replay` with `concurrent` load under-delivers
`concurrency_level`, especially with `tool_call_latency_sec`: ready
conversations wait behind busy ones, the shortfall grows with server
latency, and rollover can overlap.

Scheduling is now completion-driven. Each worker's queue token selects
the ready conversation that has waited longest instead of a fixed round.
A conversation returns to the ready pool only after its request and
simulated tool work finish, so `concurrency_level` means active
conversations, including tool time.

## Root cause

- `get_data()` generated the stage in fixed round-robin order, and
`load_lazy_data()` mapped each index directly to a conversation and
round, ready or not.
- Workers took a permit before `get_context()` waited for the preceding
turn and tool delay, so a busy conversation held capacity while a
completed one could not advance.
- Rollover replaced the session before the final old turn finished.

## Why the datagen owns readiness (review 1)

#901 fixed this class for trace replay by acquiring the permit after the
dependency wait, keeping scheduling in loadgen. I rejected that here:

- `get_context()` waits inside request materialization, after the queue
entry names a conversation. Releasing the permit frees capacity but not
the queue position, so the worker still cannot dispatch another ready
conversation.
- Choosing the next conversation needs per-slot state (round, busy,
rollover), which the datagen owns; moving it into loadgen needs a larger
API than one terminal callback.
- Counting tool latency as active time changes what `concurrency_level`
means; a permit-after-wait shape bounds HTTP in flight instead.

`on_request_finished` is terminal and idempotent, covering success,
failure, cancellation, and retired requests.

## Fix

- Per-slot scheduler and FIFO ready queue
(`conversation_replay_datagen.py:174,254`)... (continued)

85 of 89 new or added lines in 4 files covered. (95.51%)

1 existing line in 1 file now uncovered.

13521 of 15720 relevant lines covered (86.01%)

0.86 hits per line

Uncovered Changes

Lines Coverage ∆ File
2
85.89
-0.14% inference_perf/loadgen/load_generator.py
1
98.32
-0.74% inference_perf/config/config.py
1
91.49
2.1% inference_perf/datagen/replay/conversation_replay_datagen.py

Coverage Regressions

Lines Coverage ∆ File
1
79.17
-1.04% inference_perf/datagen/base.py
Jobs
ID Job ID Ran Files Coverage
1 42664894421.1 09 Nov 2026 05:38PM UTC 140
86.01
GitHub Action Run
Source Files on build 42664894421
  • Tree
  • List 140
  • Changed 5
  • Source Changed 4
  • Coverage Changed 5
Coverage ∆ File Lines Relevant Covered Missed Hits/Line
  • Back to Repo
  • Github Actions Build #42664894421
  • 6d6cbad1 on github
  • Prev Build on main (#42306871048)
  • Next Build on main (#42755379419)
  • Delete
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc