FlexMeasures / flexmeasures / 36182029517

86%

Build Type

push

github

Committed by web-flow

Commit Message

Feat/2393 durable automation runs (#2457)

* feat(data/models): add durable automation run records

An automation occurrence currently leaves no trace of its own: the only durable
marker is the automation's scheduling cursor, which says an occurrence was
attempted, not what happened to it.

Record each occurrence a runner picks up as an AutomationRun, with the attempts
made on it (AutomationRunAttempt) and the jobs it intends to create
(AutomationRunJob). Dispatch progress and worker execution outcome are tracked
separately, because 'queued everything' and 'the jobs succeeded' are different
questions an operator needs answered.

The database enforces one run per automation, occurrence and schedule revision,
and one job intent per run and logical job key. The new schedule revision on
Automation keeps runs of an edited or reactivated schedule apart from the runs
of the schedule it replaced, even at the same scheduled UTC time.

All run timestamps are validated to be timezone-aware and stored as UTC.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

* feat(data/services): claim automation occurrences durably and resume partial dispatch

The runner used to advance and commit the scheduling cursor before queueing
anything, guarded only by a Redis key with a two-minute TTL. A failure before
the first enqueue therefore lost the occurrence for good, while retrying a
partial enqueue could have duplicated work.

Claim each due occurrence into an AutomationRun instead, write down the plan for
the run before the first enqueue, and give every intended job a logical key and,
from it, a deterministic RQ job ID. A retry replays that stored plan: jobs whose
IDs are already in Redis are recognised and left alone, and only the missing ones
are queued. Because the plan holds the parameters and timings the occurrence was
planned with, a retry hours later still dispatches the occurrence as originally
intended, even if the automation has been edited sin... (continued)

Coverage Stats

462 of 487 new or added lines in 8 files covered. (94.87%)

1 existing line in 1 file now uncovered.

20055 of 23292 relevant lines covered (86.1%)

0.86 hits per line