kobotoolbox / kpi / 30269082240

82%

master: 76%

Build Type

push

github

Committed by web-flow

Commit Message

fix(longRunningMigrations): persist LRM 0027/0028 cursor DEV-2427 (#7312)

### 📣 Summary

Fixed a background data-repair job that could stall without making
progress after a restart.

### 📖 Description

Two background migration jobs (LRM 0027 and 0028) backfill a stable
identifier (`root_uuid`) for old submissions. They kept track of how far
they'd gotten with an in-memory position that reset to the very
beginning every time the job restarted, so a restart could waste a lot
of time re-checking records it had already confirmed didn't need any
work, instead of making forward progress. This position is now saved
outside the job, so a restart picks up where it left off.

### 💭 Notes

Root cause: `last_xform_id` (0027) and `last_id` (0028) were plain local
variables. Submissions belonging to XForms these jobs permanently skip
(`pending_delete`, or tagged `kobo-root-uuid-failed-0027`) never get a
`root_uuid`/`meta/rootUuid`, so they always match the "still needs
backfill" query — every restart re-scanned past the same
permanently-skipped records before reaching new work, which could
prevent either job from ever completing a full pass in practice.

Both cursors are now persisted in Redis (`cache`, no TTL, cleared on
natural completion). For 0027, the cursor only advances once every XForm
in the current batch has reached a terminal outcome, so a crash
mid-batch re-processes that small batch on restart instead of
permanently skipping whatever wasn't finished. For 0028, the cursor
advances right after each batch is processed, same as before.

Also excludes trashed XForms from LRM 0028's Postgres lookup
(`xform__pending_delete=False`), for symmetry with LRM 0027's own
exclusion.

Coverage Stats

9319 of 12762 branches covered (73.02%)

0 of 13 new or added lines in 2 files covered. (0.0%)

142 existing lines in 17 files now uncovered.

30333 of 37483 relevant lines covered (80.92%)

5.52 hits per line