akvo / akvo-mis / #496

88%

push

coveralls-python

Akvo Flow Data Seeding (#149)

* [#148] feat: add .gitignore files for caddisfly, data, invalid_values, and seeded directories

* [#148] feat: Add robust Akvo Flow data seeder management command

Add a full-featured Django management command flow_data_seeder.py to
import Akvo Flow CSV exports into MIS FormData/answers.
- Provide CLI args: `--form` (required), `--limit`, and `--revert`.
- Validate inputs and print clear console messages for success/error.
- Load question mappings from `<FLOW_SOURCE_DIR>/{form}_forms_mapping.csv`.
- Load Flow data files from `<FLOW_SOURCE_DIR>/data/{form}_*.csv`.
- Load administration mappings from `<FLOW_SOURCE_DIR>/administration_mapping.csv`.
- Pre-fetch MIS Questions in a single DB query to avoid N+1 issues.
- Extract and normalize Flow values (JSON/text/number/date/base64).
- Handle special types:
    - Caddisfly results: extract numeric and flatten result rows for CSV debug.
    - Options/multiple options: map labels to option values.
    - Geo, photo, signature, number, administration handling.
- Collect invalid values and provide optional CSV export for debugging (`invalid_values`).
- Optionally write debug CSVs: `caddisfly`, `invalid_values`, and `seeded`.
- Insert data transactionally with `transaction.atomic()` and bulk-create answers.
- Implement a `--revert` flow that reads seeded CSV and deletes created `FormData` records.
- Improve maintainability with constants, type hints, docstrings, and structured helper methods.
- Provide robust error handling and informative logging for reviewers and operators.

fix: Correct field name for created_at in FormData creation and update results handling logic

* [#148] feat: Add comprehensive test suite for flow_data_seeder management command

* [#148] feat(flow-seeder): robust Akvo Flow data seeding with mapping, validation, and bulk insert

Add constants and structured column/key names for maintainability and readability.
Implement robust CSV loading and file-globbing ... (continued)

3694 of 4296 branches covered (85.99%)

Branch coverage included in aggregate %.

7617 of 8528 relevant lines covered (89.32%)

0.89 hits per line