OCHA-DAP / hdx-scraper-insecurity-insight / 22501904795

push

github

Final refactor (#8)

* Align to cookiecutter structure

* Move functions out of main

* Remove unneeded variables

* Add env back to tests

* Add retriever

* Move dataset metadata to config

* Remove additional file

* Reorganize

* Rewrite fetch_api_response

* Rewrite fetch_datasets

* Rewrite check_api

* Clean up variables

* Update fresh data check

* Delete files

* Rewrite refresh spreadsheets

* Update metadata in config to match HDX

* Update tags

* Begin changes to create_dataset

* Add test data

* Formatting

* Rewrite update_datasets

* Add try/except for download

* Add dataset files

* Update saved data

* Clean up utilities

* Fix date selection

* Add optional countries parameter

* Remove old tests

* Add another optional countries parameter

* Rewrite tests

* Update readme

* Remove env variables from test run

* Rename files

* Rename again

* Change main parameters

* Add check for missing endpoints

* API agnostic (#5)

* Remove check against API

* Remove schema doc

* Remove call to deleted function

* This runs to completion creating datasets but simply merges crsv and svpoliticalviolence which isn't what Insecurity Insight are after

* Runs but test not updated

* Fixes for country CRSV and SVPoliticalViolence datasets

* Update metadata
Handle no country iso by using country name
Fix min max date calculation

* Refactor spreadsheet gen pt 1

* Use pandas for filtering

* Update test

* Update test

* Remove topics and force_refresh

* Change food security to fivc and update country topics

* Add country overview resources

* Remove extra dash in file names

* Add comment

* Add function to delete old API resources

* If current year not in data use previous

* Update requirements

* Update requirements

* Fix dates

* Fix start dates and water names

* Fix ISR dataset

* Fix numeric values

* Fix numeric values (2)

* Fix numeric values (3)

* Cap dates at today

* Censor all geo information

* Update start dates

* Fix r... (continued)

368 of 428 new or added lines in 4 files covered. (85.98%)

368 of 428 relevant lines covered (85.98%)

0.86 hits per line