run-llama / llama-agents / 37945987730

85%

push

github

Rewind in-progress work inside the reducer at every session start (#783)

## What changed

Replaying a tick journal that spans a resume produces the wrong broker
state. Concretely: a run is interrupted while step `A` is in progress on
worker slot 1, then resumed. On resume the runner rewinds `A` back to
the queue and re-dispatches it, and the rewind hands out the smallest
free slot, so the rerun gets slot 0. The second session's
`TickStepResult` is journaled with `worker_id=0`. Replaying that journal
from the start folds session 1 (slot 1 in progress), reduces the
`TickSessionStart` for session 2 without rewinding, and then reduces a
result for slot 0 that does not exist. That raises `ValueError: Worker 0
not found in in_progress` from `_find_in_progress`, and with two
interrupted workers it does not raise at all: the rewind re-queues them
in reverse order, so the result for slot 0 is applied to the other
worker's event.

The runner used to rewind outside the reducer at every resume, and the
replay helpers imitated that by rewinding once, before folding, no
matter how many sessions the journal held. Every session already
journals `TickSessionStart` first. The reducer now performs the rewind
when it reduces that tick, using the tick's own stamp as the clock, and
returns the wakeup commands the rewind schedules so the runner processes
them like any other command. The runner's separate rewind call is gone.
One fold over a journal with three sessions now rewinds three times, at
the same points the live run did, and reproduces its worker ids.

## Who hits this

- The server persistence runtime rebuilds state by replaying its stored
journal on startup. The journal spans sessions after the first restart,
so the second restart of a run that was interrupted mid-step replays
into the error above.
- `ctx.to_dict()` rebuilds broker state from the in-memory tick list on
every call, so a context resumed once and snapshotted again had the same
problem on `BasicRunti... (continued)

2 of 2 new or added lines in 1 file covered. (100.0%)

18495 of 21776 relevant lines covered (84.93%)

1.14 hits per line