kobotoolbox / kpi / 25750860804

82%

master: 76%

push

github

fix(subsequences): migrate qpath to xpath in SubmissionSupplements DEV-2094 (#7046)

### 📣 Summary
Ensure old NLP data and QA question responses get exported.


### 📖 Description
Fixes a bug wherein older transcripts, translations, and qa question
responses were not showing up in exports or the data table.


### 💭 Notes
A previous migration from `qpath`s to `xpath`s was done on the fly, so
submission supplements that were not accessed were not updated. However,
the `advanced_features` dict of the corresponding assets were updated to
have xpaths, and no reference to the old qpaths.

The subsequent migration to the new subsequence version (`20240820`)
looked at old submission supplements
to gather `(xpath, action)` pairs to determine what QAFs needed to be
created, then used the `advanced_features` dict on the asset to populate
the `params` of the new QAF. This led to the creation of QAFs that had a
qpath instead of an xpath in `question_xpath` and an empty `params`
list.

This PR queries all submission supplements for anything with a hyphen in
the question name using `extra` and `rawSQL`. This is because there is
no Django-native way of querying for keys in a JSONField that match a
particular regex. While this does mean using postgres specific code, it
prevents us from having to use a whole new schema version for what is
essentially an edge case.

The PR also creates any missing QAFs that may have not been created if
every submission supplement for an asset used the old qpath. For safety,
the QAFs are created in an `atomic` block while updating the submission
supplement that necessitated them, so if something kills the update
mid-process, we will still know they need to be created. We use a simple
set to keep track of which QAFs were already created so we don't hit the
DB more than necessary.

There is one small risk where we could end up with the migration
infinitely in progress:
If there is a submission supplement with a qpath that has no xpath
equiv... (continued)

7800 of 12051 branches covered (64.72%)

1 of 98 new or added lines in 2 files covered. (1.02%)

29618 of 36161 relevant lines covered (81.91%)

5.77 hits per line