kobotoolbox / kpi / 22070819676

81%

master: 76%

push

github

fix(hooks): prevent missing logs and add worker failure recovery DEV-1728 (#6725)

### 📣 Summary
REST Services hooks now reliably create logs for all submissions and
recover from Celery worker interruptions, preventing silent failures and
data loss.

### 📖 Description
Two new Celery periodic tasks have been added to automatically recover
from REST Service processing failures.

The first task, `retry_stalled_pending_submissions`, detects submissions
that were queued but never processed due to worker failures. It
automatically re-queues submissions that have been waiting for more than
2 hours without any processing attempt. This ensures submissions
eventually reach their external endpoints even after worker crashes. You
can monitor its activity by checking Celery logs for messages like
"Re-queued X stalled submissions".

The second task, `mark_zombie_processing_submissions`, detects
submissions that started processing but were interrupted mid-flight.
When processing takes longer than 2 hours, it marks those submissions as
failed, indicating the worker was likely killed during execution. It's
important to note that these submissions may have already reached the
external endpoint before the interruption occurred. The failure message
advises users to verify manually on the remote server to avoid duplicate
submissions. Monitor this task through logs showing "Marked X zombie
submissions as failed".

Both tasks use the `HOOK_PROCESSING_TIMEOUT` setting (default: 120
minutes) and run periodically as defined in `CELERY_BEAT_SCHEDULE`. No
additional configuration is required as they work out of the box.

7240 of 11333 branches covered (63.88%)

92 of 103 new or added lines in 9 files covered. (89.32%)

27865 of 34243 relevant lines covered (81.37%)

1.6 hits per line