OCHA-DAP / hdx-scraper-ophi / 33691633711

92%

main: 97%

push

github

Fix two live-run bugs found while UI-testing the Dagster pipeline

Neither surfaced under test_dagster_defs.py's materialize() call, only
under a real dagster job launch:

- RetrieverResource.setup_for_execution() never set the global user agent,
  so any Download() call outside the HDX API (i.e. every OPHI/boundary
  download) raised UserAgentError as soon as a real HDX Configuration
  singleton existed. hdx.facades.simple.facade does this assignment for
  the same reason.
- ophi_core_job's default multiprocess executor gives each asset its own
  subprocess and thus its own RetrieverResource instance, but the resource
  is designed for one shared scratch folder/batch across the whole run
  (see its docstring) - concurrent assets raced on that shared folder,
  with one asset's teardown deleting it while another was still writing.
  Pinned the job to in_process_executor, matching how materialize() in
  tests already runs it.

2 of 3 new or added lines in 1 file covered. (66.67%)

497 of 530 relevant lines covered (93.77%)

0.94 hits per line