OCHA-DAP / hdx-scraper-ophi / 33713061739

92%

main: 97%

push

github

Add a country_results_summary_job, Dagster's equivalent of Airflow's
end-of-run summary task

Unlike Airflow, each country_mpi_dataset partition materializes as its
own separate run (via ophi_country_fanout_sensor), not a step within one
run - so there's no single run whose task graph a summary step could sit
downstream of the way Airflow's DAG allows. This is instead a standalone
job (summarize_country_results op) meant to be run manually or on a
schedule once the fan-out is believed complete, not auto-chained.

It finds the latest ophi_country_job run for every registered country
partition via RunsFilter(tags={"dagster/partition": iso3}), counts
outcomes, and for any FAILURE run fetches the failure message plus the
full step compute log via the instance's own ComputeLogManager - the
same "everything visible in one place" result as Airflow's version, but
using Dagster's own APIs instead of reading log files directly, so it
works with any configured backend (local, or e.g. AzureBlobComputeLogManager
in a real deployment), not just local disk.

Verified with a real induced STEP_FAILURE in a test (execute_in_process,
not materialize(), since RunsFilter needs runs tagged with the actual
"ophi_country_job" name) - the summary correctly reported the failure
count and inlined both the error message and the full compute log.
Also smoke-tested live against the running dev instance.

38 of 41 new or added lines in 1 file covered. (92.68%)

550 of 599 relevant lines covered (91.82%)

0.92 hits per line