OCHA-DAP / hdx-scraper-ophi / 33709626546

94%

main: 97%

push

github

Add an end-of-run summary task for the country fan-out

country_mpi_dataset now returns a small result dict ({countryiso3, status,
folder}) instead of just the scratch folder. A new summarize_country_results
task (trigger_rule=all_done, downstream of the whole country_mpi_dataset
mapped group) pulls every mapped instance's XCom result and logs one
consolidated per-status count, plus the list of countries that didn't
report a result at all (failed or still running).

For those missing countries it goes further and inlines their mapped
instance's full log file content directly into this one task's own log,
read straight off disk - this only works for the default local-file log
handler on a self-hosted single-machine setup like this one, not a remote
logging backend or a distributed executor with no shared filesystem, but
it fully closes the "look in 109 places" gap for that case: verified live
against a real run with genuine failures (Google Sheets 409s and the
known HDX-stage 429 rate-limiting), and every failing country's exact
error was visible in this one task's log without opening any of the
individual per-country task logs.

27 of 32 new or added lines in 1 file covered. (84.38%)

527 of 559 relevant lines covered (94.28%)

0.94 hits per line