brawer / osmdiffs / 38139035579

94%

push

github

refactor(pipeline): Fetch all inputs up front, in parallel (#831)

The pipeline now starts with a single `fetch_inputs` step that
downloads AllThePlaces and the OpenStreetMap planet concurrently
(sub-steps `fetch_inputs.atp` / `fetch_inputs.osm`). If one fetch
fails, the others are cancelled and the run aborts before any
processing. `import_atp` and `import_osm` now work only on local
files.

- `fetch_planet` is async (no nested runtime) and hashes the planet
  while streaming, including the already-downloaded prefix of a
  resumed `.part` file, instead of re-reading ~90GB afterwards.
- Cache rule, documented in `pipeline::inputs`: an input is reused
  when its metadata sidecar exists together with either the raw file
  or the artifact derived from it (`alltheplaces.parquet`,
  `osm-features.index`). A sidecar-less `.pbf` (regional extract) is
  still accepted.
- The release date is derived right after `fetch_inputs`, from the
  fetched metadata.
- Cancelled steps still log an `end` snapshot plus a warning.

Output is unchanged: on the fixtures, conflated.parquet, the tiles,
datapackage.json and the BOM are byte-identical to the previous
version.

Closes #830


Claude-Session: https://claude.ai/code/session_01WtzCoNtkdZsv5JqAkTcU6j

Co-authored-by: Claude <noreply@anthropic.com>

430 of 440 new or added lines in 7 files covered. (97.73%)

10903 of 11671 relevant lines covered (93.42%)

1330.3 hits per line