FlexMeasures / flexmeasures / 37698722666

87%

main: 87%

Build Type

push

github

Committed by web-flow

Commit Message

Drop a belief only when it repeats the belief right before it (#2687)

* perf: compare new beliefs with stored ones in one pass

drop_unchanged_beliefs ran a groupby.apply per belief, about 7 ms each, on every
re-computed schedule and re-posted measurement; a week of 15-minute beliefs took 5 s.

Signed-off-by: Stijn van Houwelingen <teadrinkingprogrammer@proton.me>

* docs/changelog: link PR #2639 in the beliefs entry

Signed-off-by: Stijn van Houwelingen <teadrinkingprogrammer@proton.me>

* data/services: drop a belief only when it repeats the belief right before it

Context:
- #2686: when one save holds several belief times for the same event, a
  value that changes and changes back loses its last belief, so a lookup of
  the most recent belief returns a value the source had already moved away
  from. Found by Belhsan in #2685.
- The in-frame pass deduplicated on (event, source, probability, value),
  without belief time, so it dropped a value seen anywhere earlier in the
  frame. The stored-belief pass compared with the latest stored belief only,
  blind to earlier rows of the same save.
- Both date from 35591a384 (Nov 2022), which began accepting such frames
  instead of refusing them; #1918 changed which wrong answer one of the
  combinations gave.

Change:
- Line up stored and new beliefs about each event from each source in
  belief-time order, and drop a new belief only if it says the same as the
  belief right before it. A stored belief counts as coming right before a new
  one at the same belief time, so exact duplicates are still dropped, and a
  different value there is left to FLEXMEASURES_ALLOW_DATA_OVERWRITE.
- Compare beliefs whole: a probabilistic belief is unchanged only if every
  probability keeps its value, and is kept whole otherwise. The in-frame pass
  used to keep only the probabilities whose values changed.
- One comparison over the whole frame replaces a groupby-apply per belief,
  so the pandas level-name workaround for tha... (continued)

Coverage Stats

46 of 46 new or added lines in 1 file covered. (100.0%)

1 existing line in 1 file now uncovered.

21182 of 24494 relevant lines covered (86.48%)

0.86 hits per line