FlexMeasures / flexmeasures / 38236066248

87%

push

github

fix: Count automation jobs in a fixed number of Redis round trips (#2647)

* Read a job cache's entries in a fixed number of Redis round trips

JobCache.get pinged Redis, read the entry's job IDs with SMEMBERS, and then
fetched each job with its own Job.fetch. A page that reads many entries, such
as the jobs of every sensor in an asset's subtree, so paid a round trip per
entry and another per job, which adds up quickly once Redis is not on the
same machine.

JobCache.get_many reads any number of entries in a fixed number of round
trips: one ping, one pipeline for the job IDs of all entries, one
Job.fetch_many for all jobs (fetching a job listed under several entries
once), and one pipeline to remove the IDs of jobs that expired. JobCache.get
is now a single-entry call to it, so its behaviour is unchanged.

The two mock-based tests of get stubbed Job.fetch, so they are replaced by
tests on fakeredis with real jobs: one checks that expired jobs are left out
and removed from the cache, and one pins the number of direct commands and
pipelines that get_many sends.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

* Count an automation's jobs in a fixed number of Redis round trips

The Automations page shows, per automation, how many of its jobs are queued,
finished or failed, and refreshes this every minute. Counting them read each
job cache entry separately, and then asked Redis for every job's status
again with get_status(), although the job had just been fetched with its
status. With 1000 jobs that is over 2000 round trips per refresh, per open
tab.

The counts now read all entries with JobCache.get_many and take each job's
status from the fetched job with get_status(refresh=False). On a test site
with 22 sensors and 1000 jobs, this brings the automations list endpoint
from 1.1 s to 0.35 s with Redis on the same machine, and from 2.0 s to
0.34 s at 0.6 ms per round trip, with identical responses.

A test counts the Redis commands an... (continued)

28 of 28 new or added lines in 2 files covered. (100.0%)

21419 of 24722 relevant lines covered (86.64%)

0.87 hits per line