OCHA-DAP / hdx-scraper-ophi / 33691640148

94%

main: 97%

push

github

Fix three live-run bugs found while UI-testing the Airflow pipeline

None of these surfaced under test_airflow_defs.py, which calls the task
functions directly rather than going through a real Airflow scheduler run:

- retriever_context() never set the global user agent (same root cause and
  fix as the Dagster RetrieverResource bug).
- delete_scratch_on_success defaulted to True, but several tasks in this
  DAG run concurrently against the one shared scratch folder (the four
  downloads, and the whole country_mpi_dataset fan-out) - one task
  finishing first deleted the folder while others were still using it.
  Defaulted to False instead; Dagster's equivalent race is fixed by
  pinning its job to in_process_executor, not an option here since every
  Airflow task is already its own process. Cleanup is left out of scope
  for this local comparison.
- is_target_run_day(logical_date) had no default, but Airflow only adds
  "logical_date" to the task context at all for scheduled/backfill runs -
  a manual trigger with no date omits the key entirely rather than passing
  None, so the task failed with a missing-argument TypeError. Added a
  None default (manual runs now always execute, matching a manual Dagster
  launch bypassing ophi_annual_schedule's SkipReason check entirely).

Verified end-to-end via a real `airflow standalone` run: the annual
schedule's skip logic fired correctly on a non-target scheduled tick, and
a manual trigger ran the full DAG - 7 core tasks plus 99/109 country_mpi_
dataset fan-out instances succeeded (the other 10 hit real HDX-stage-API
429 rate limiting from ~30 concurrent LocalExecutor workers hitting it at
once, not a code bug - see the plan's comparison notes).

5 of 6 new or added lines in 2 files covered. (83.33%)

497 of 526 relevant lines covered (94.49%)

0.94 hits per line