FlexMeasures / flexmeasures / 30824372247

83%

main: 84%

push

github

Summarise which sources recorded for which sensors

Two mirrored accessors ask which data sources have recorded for which sensors:
Sensor.data_sources reads timed_belief in sensor_id order,
DataSource.sensors reads it in source_id order.
Both answer a question about a relation bounded by sensors times sources,
a few thousand rows at most,
by reading the largest table in the database.

Reordering the primary key to lead with sensor_id (d4a7c1e93b52) makes the second one
worse: nothing leads with source_id any more, so it degrades to a sequential scan.
An index on source_id would fix that, but at a large fraction of what that migration
reclaims, to answer a question about a few thousand rows.

Add a sensor_data_source summary table instead, keyed on the pair.
It serves both accessors as a plain lookup,
so it also removes the sensor-leading one's dependence on scanning beliefs.

Maintained by a statement-level trigger rather than from the save path.
That was not the first design: the hook originally lived in TimedBelief.add_to_session,
on the reasoning that every save funnels through it.
Two existing tests then failed with empty results,
because they insert beliefs without going through that path -- and so do bulk inserts,
COPY, plugins and raw SQL.
A hook would have left the summary silently incomplete in exactly the cases nobody
checks, so the trigger does it instead: unbypassable whatever the insert route,
and one small upsert per statement rather than per row.
A test inserts through raw SQL specifically to pin that.

The trigger and its function are created both by the migration and by a create_all
DDL listener, so a schema built either way behaves the same.
Both are written idempotently, since create_all runs once per test module.

The summary is deliberately a superset:
pairs are added on insert and not removed on delete,
because deciding whether a pair went stale needs exactly the scan this avoids.
Sensor.search_data_sources still reads timed_beli... (continued)

11 of 12 new or added lines in 2 files covered. (91.67%)

16603 of 19892 relevant lines covered (83.47%)

0.83 hits per line