kobotoolbox / kpi / 29743815627

81%

master: 76%

Build Type

push

github

Committed by web-flow

Commit Message

feat(longRunningMigrations): keep migration lock fresh with a heartbeat DEV-2427 (#7270)

### 📣 Summary

Background data migrations can now run for up to 24 hours per pass and
recover on their own if a server is restarted, so they finish more
reliably.

### 📖 Description

Some database maintenance tasks run in the background over a very large
amount of data. Until now each pass was capped at about 71 minutes, and
if the server running one was restarted it could stay stuck for a long
time before another server picked it up. These tasks now get up to 24
hours per pass and are retried within minutes if their server goes away,
so they complete without manual intervention.

### 👷 Description for instance maintainers

Long-running migrations now get a dedicated, much longer time budget, up
to about 24 hours per run, separate from the shared limits used by other
background tasks (those keep their current values). New environment
variables let you tune this:
`CELERY_LONG_RUNNING_MIGRATION_TASK_TIME_LIMIT`,
`CELERY_LONG_RUNNING_MIGRATION_TASK_SOFT_TIME_LIMIT`,
`CELERY_LONG_RUNNING_MIGRATION_TASK_HEARTBEAT_INTERVAL` and
`CELERY_LONG_RUNNING_MIGRATION_TASK_HEARTBEAT_TTL`.

While a migration runs, a background heartbeat keeps its lock alive with
a short lease and records that it is still making progress. If the
worker running it dies (for example a Kubernetes pod eviction), the
lease expires within a few minutes and the migration is picked up again
automatically, instead of being blocked until the old time limit
elapsed.

No database schema change and no new migration to run. The defaults are
safe, so no configuration change is required.

Coverage Stats

9304 of 12654 branches covered (73.53%)

5 of 27 new or added lines in 2 files covered. (18.52%)

751 existing lines in 36 files now uncovered.

30322 of 37254 relevant lines covered (81.39%)

5.51 hits per line