FlexMeasures / flexmeasures / 29926316522

83%

Build Type

push

github

Committed by web-flow

Commit Message

Speed up post-processing of sensor data searches (#2328)

* perf: vectorize keep_latest_version

Replace the temp-column + sort + drop_duplicates + index-isin implementation
with vectorized groupby transforms over integer arrays. Versions are now
parsed once per unique source instead of once per row, and a fast path
returns immediately when every source is the sole member of its
(name, type, model) group, i.e. when nothing can be dropped - the common
case, decided without touching the data.

Behavioral notes:
- With one_deterministic_belief_per_event=True and multiple beliefs per
  event from the winning source, all its beliefs are now kept (downstream
  code picks one); previously an arbitrary one survived, as the sort was
  unstable.
- Columns named source.* in the input frame are no longer dropped.

Also add a randomized equivalence test against a naive reference
implementation.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* perf: fast track deterministic multi-source belief selection per event

When one_deterministic_belief_per_event is requested on deterministic data
(probabilistic depth 1), taking the median per belief is a no-op, so skip
the per-belief groupby-apply entirely and pick the winning belief per event
with a single vectorized lexsort (latest source version first, most recent
belief time second - matching the existing sort; the code comment claiming
belief time takes precedence was stale).

For the remaining probabilistic slow path, parse each unique source version
once instead of once per row, and replace the per-event groupby-apply with
a duplicated-mask on the sorted frame.

Add a DB-free randomized equivalence test against a naive reference.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* perf: vectorize compressed chart-data serialization

Replace the per-row iterrows loop in Sensor.search_beliefs' compress_json
branch with column-wise epoch conversions and a single tight loop over
plain Python scalars, factored into a compr... (continued)

Coverage Stats

96 of 104 new or added lines in 5 files covered. (92.31%)

2 existing lines in 1 file now uncovered.

15786 of 19093 relevant lines covered (82.68%)

0.83 hits per line