FlexMeasures / flexmeasures / 39836790812

87%

push

github

Refuse duplicate data sources, merging existing ones and their references (#2613)

* Reuse the oldest data source when duplicates exist

Concurrent calls to get_or_create_source can each insert the same source,
because the unique constraint on data_source treats NULL user and account IDs as distinct.
Every later lookup then raised MultipleResultsFound, failing all schedules under that configuration (#2611).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* Share one job between concurrent identical calls to a job-cached function

The job cache checked for a cached job, and only stored a new one after creating it,
so two identical calls arriving together both missed the cache and each created their own job.
For scheduling, the two jobs then each created their own data source for the same configuration (#2611).
Hold a Redis lock on the hash of the call's arguments while checking the cache and creating the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* Add changelog entry for PR #2612

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* Read the cached job ID once, as the key can expire between two reads

Checking that the job cache key exists and then getting it could return None if the key expired in between,
failing on .decode().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* Name the duplicate source IDs in the warning, and empty the queue after the concurrency test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* Join the concurrency test's threads with a timeout, and stop shadowing the source argument

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>

* Refuse duplicate data sources, after mergi... (continued)

91 of 95 new or added lines in 4 files covered. (95.79%)

22338 of 25700 relevant lines covered (86.92%)

0.87 hits per line