kobotoolbox / kpi / 38784352033

83%

master: 76%

Build Type

push

github

Committed by web-flow

Commit Message

fix(attachments)!: recount drifted storage counters DEV-3015 (#7696)

### 📣 Summary
Storage usage that drifted away from the files actually stored,
sometimes below zero, gets repaired.

### 📖 Description
#7680 stopped storage usage from drifting, but did not repair the
accounts where it had already drifted. Their projects are now recounted
from the files they actually hold, in the background, so their storage
usage matches what they really store again. Some accounts may then show
more storage than before, and others less.

### 👷 Description for instance maintainers
A new partial index on attachments lets the recount find the projects
that may have drifted quickly. With `SKIP_HEAVY_MIGRATIONS`, migration
`logger/0056` only prints the SQL to run by hand:

```sql
CREATE INDEX CONCURRENTLY "attach_inactive_xform_id_idx"
    ON "logger_attachment" ("xform_id")
    WHERE "delete_status" IS NOT NULL;
```

A new periodic task, `recount-drifted-storage-counters`, recounts these
projects in turns and starts over once done.

Storage recounts no longer suspend submissions. The
`fix-stale-submissions-suspended-flag` task, which unlocks accounts left
suspended (e.g. by a worker of the previous version stopped during the
deployment), now runs every 15 minutes on the low priority queue instead
of on Sunday nights only.

⚠️ **Known issues:** Submission suspensions can be lifted while
`populate_submission_counters` recalculates user's counters. This will
be fixed in another PR.

⚠️ **Breaking change**: in addition to what #7680 says about it,
`update_attachment_storage_bytes` is rewritten. It only recounts project
counters, from the attachments they actually hold, without suspending
submissions. `--sync`, `--no-lock` and `--chunks` are gone, and
`--force` now means "start over". Usage:
- `./manage.py update_attachment_storage_bytes`: recount the projects
that may have drifted (same selection as the task)
- `--username <username>`: recount every project of one user
- `-... (continued)

Coverage Stats

10369 of 13687 branches covered (75.76%)

136 of 142 new or added lines in 6 files covered. (95.77%)

6 existing lines in 2 files now uncovered.

33149 of 39878 relevant lines covered (83.13%)

5.51 hits per line