open-contracting / data-registry / 16084075787

push

github

fix: Summarize OCDS Merge warnings, as they are too large to store

Used this script to manually migrate the notes:

import re
from collections import Counter
from data_registry.models import Job
OCDS_MERGE_WARNING_PATTERN = re.compile(
    r"^Multiple objects have the `id` value '.+' in the `(.+)` array$", re.MULTILINE
)
for job in Job.objects.exclude(process_notes={}).defer("process_notes"):
    counts = Counter()
    others = []
    for note, data in job.process_notes['WARNING']:
        # OCDS Merge has one warning type, which can be issued millions of times.
        if note.startswith("Multiple objects have the `id` value "):
            counts.update(OCDS_MERGE_WARNING_PATTERN.findall(note))
        else:
            others.append([note, data])
    if not counts:
        continue
    for path, count in counts.items():
        others.append(["OCDS Merge", {"count": count, "path": path}])
    job.process_notes['WARNING'] = others
    job.save(update_fields=["modified", "process_notes"])
    print(job)

5 of 24 new or added lines in 2 files covered. (20.83%)

959 of 1645 relevant lines covered (58.3%)

0.58 hits per line