kobotoolbox / kpi / 24132020822

82%

master: 76%

Build Type

push

github

Committed by web-flow

Commit Message

perf(beat): throttle Beat schedule reloads to fix dispatch starvation DEV-1963 (#6926)

### 👷 Description for instance maintainers

On high-volume servers (Global), Celery Beat was spending most of its
time reloading its schedule instead of dispatching tasks. Root cause:
each completed trash-bin task deleted its `PeriodicTask` record
individually, triggering a `PeriodicTasks.changed()` DB signal that
forces Beat to reload all periodic tasks from scratch. With hundreds of
concurrent one-off tasks completing, this produced thousands of reload
signals per minute.

Two changes fix this:

1. **`ThrottledDatabaseScheduler`** (`kobo/apps/beat/schedulers.py`) — a
`DatabaseScheduler` subclass that gates full schedule reloads to at most
once every 15 seconds. Added as `CELERY_BEAT_SCHEDULER` in settings and
as `--scheduler` in the Beat entrypoint.

2. **Removed individual `PeriodicTask.delete()` calls** from
`process_deletion` (`trash_bin/utils/trash.py`) and the
`pre_delete_attachment` signal (`openrosa/apps/logger/signals.py`).
Orphaned `PeriodicTask` records are now cleaned up in batch by the
existing `garbage_collector` task (runs every 30 min) with signals
suppressed via `temporarily_disconnect_signals`.

### 💭 Notes

- `ThrottledDatabaseScheduler` always calls `super().schedule_changed()`
to keep `_last_timestamp` in sync with the DB — no change is ever
permanently missed, at most deferred by up to 15 s. For trash tasks
scheduled days in the future this is negligible.
- Orphaned `PeriodicTask` records are `one_off=True` and already
`enabled=False` (Beat marks them after dispatch). Beat loads only
`.objects.enabled()`, so orphans don't trigger any extra reload signals
while waiting for `garbage_collector`.
- Beat's own internal signal when it marks `one_off` tasks as
`enabled=False` after dispatch (1 per dispatched task) is still present
— it's internal to `django-celery-beat` and not addressable without
patching the library. The throttle absorbs this rema... (continued)

Coverage Stats

7441 of 11560 branches covered (64.37%)

3 of 25 new or added lines in 4 files covered. (12.0%)

1 existing line in 1 file now uncovered.

28568 of 34968 relevant lines covered (81.7%)

5.81 hits per line