kobotoolbox / kpi / 26871739809

82%

master: 76%

Build Type

push

github

Committed by web-flow

Commit Message

feat(bulkProcessing): implement background execution and polling for bulk actions DEV-1494 (#7080)

📣 Summary
Users can now successfully run bulk transcription and translation jobs
in the background, with real-time progress tracking and automatic
recovery if a network interruption occurs.

📖 Description
This PR implements the core background processing engine for
SubsequenceBulkAction using Celery. It ensures that large-scale audio
processing and translation tasks run asynchronously without blocking the
main API threads.

**Key Architectural Highlights:**
* **Job Orchestration:** Added `start_bulk_item_job` to safely delegate
individual submissions to the existing
`SubmissionSupplement.revise_data()` flow, ensuring complete
compatibility with existing NLP schemas.
* **Progress Polling:** Implemented `update_batch_status` utilizing
pessimistic database locking (`select_for_update(skip_locked=True)`) to
safely calculate and update the parent job's overall progress percentage
without triggering deadlocks.
* **Feature Auto-Provisioning:** Updated the creation serializer to
automatically provision the necessary `QuestionAdvancedFeature` (the
backend column) if it does not already exist for the requested language.
* **System Resilience:** Introduced a `resume_stuck_bulk_actions`
watchdog task (run every 5 minutes) to automatically resume polling for
jobs that experience worker restarts, timeouts, or pod terminations.
* **Async Guard Fix:** Updated the execution guard in `base.py` to
ensure long-running Google Cloud operations properly trigger the
downstream polling mechanism without getting trapped in an infinite
`in_progress` state.
* `BULK_ACTION_STATUS_POLL_INTERVAL` (Default: `30` seconds): Determines
how frequently the background task queries individual child submissions
to compute the overall parent batch progress percentage.
* `BULK_ACTION_STUCK_THRESHOLD` (Default: `300` seconds / 5 minutes):
The expiration window after which an active bulk action j... (continued)

Coverage Stats

8981 of 12201 branches covered (73.61%)

73 of 459 new or added lines in 9 files covered. (15.9%)

563 existing lines in 36 files now uncovered.

29482 of 36149 relevant lines covered (81.56%)

5.56 hits per line