FlexMeasures / flexmeasures / 42307474381

88%

Build Type

push

github

Committed by web-flow

Commit Message

Speed up scheduling problem preparation for flex-models with many devices (#2516)

* Read each commitment's constant columns once, while splitting them

The scheduling problem carries commitments as DataFrames whose name, class,
commodity and deviation prices are the same value repeated down every row.
Building the model read those back one `df[column].iloc[0]` at a time:
pandas constructs a Series and positionally indexes it for each, costing a
few microseconds where a dict lookup costs a tenth of one.

That is cheap until the flex-model has a device per charging session. A site
with 20 charge points of 2 EVSEs each produces roughly 7,400 sub-commitments
per schedule, because a commitment is split into one sub-commitment per
group and each time step is usually its own group. The reads then run into
the tens of thousands per solve.

The constants are now gathered in convert_commitments_to_subcommitments,
which already has the parent frame in hand, and carried on SchedulingProblem
as `commitment_scalars`. Gathering them there rather than from the split
result is the point: the parent is read once, not once per time step, and
the group prices come from one vectorized `groupby.first()` instead of a
lookup per group.

Measured on a week of EV charging (20 charge points of 2 EVSEs each, PV,
building and battery), profiled, one simulated day:

  _validate_commitments_are_enforceable   1.79s -> 0.02s
  _price_of (59,176 calls)                2.64s -> 0.01s
  aggregate_commodity_costs               0.98s -> 0.00s
  convert_commitments_to_subcommitments   2.10s -> 2.39s  (now builds them)
  prepare_scheduling_problem             11.11s -> 9.77s
  pandas self-time                       26.73s -> 20.25s

Unprofiled, that is 24.6s -> 20.8s for a simulated day and 172s -> 138s for
the week, with the resulting schedules unchanged.

_identify_commitment and aggregate_commodity_costs now take the gathered
constants. aggregate_commodity_costs still accepts frames, whi... (continued)

Coverage Stats

143 of 166 new or added lines in 3 files covered. (86.14%)

2 existing lines in 1 file now uncovered.

23271 of 26481 relevant lines covered (87.88%)

0.88 hits per line