FlexMeasures / flexmeasures / 30828312771

83%

main: 84%

Pull #2382

github

Install the trigger before backfilling, so concurrent inserts are not lost

The migration backfilled sensor_data_source and only then created the trigger,
both inside Alembic's transaction.
That loses data.
The backfill reads a snapshot taken when its statement begins,
so beliefs inserted while it runs are not in it,
and a trigger created afterwards in the same transaction is not visible to the
sessions doing that inserting either,
so nothing ever records those pairs.
On a large table the backfill takes minutes, so the window is not theoretical.

Create the trigger first and commit it, then backfill.
The two then overlap rather than leaving a gap between them,
and ON CONFLICT DO NOTHING absorbs the overlap.
Because the steps now commit as they go, a run that fails during the backfill leaves
the table and trigger behind, so creating the table is guarded and a retry gets past it.

Demonstrated both ways by stepping two connections explicitly,
with the second inserting a new pair between the backfill and the trigger creation.
The first attempt used a sleep and failed to reproduce the loss,
which would have made the problem look theoretical;
sequencing the connections instead shows the old order dropping the pair and the new
order keeping it.

Also merges the base branch, whose only conflict was the changelog:
this branch still carried the pre-correction "three indexes" wording alongside its own
entry, so the resolution keeps both entries with the base's wording.

flexmeasures/data: 256 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>

19 of 22 new or added lines in 2 files covered. (86.36%)

16611 of 19902 relevant lines covered (83.46%)

0.83 hits per line