brawer / osmdiffs / 43022531526

94%

Build Type

push

github

Committed by web-flow

Commit Message

feat(atp): Count what happens to each spider's features (#927)

* feat(atp): Count what happens to each spider's features

import_atp now writes alltheplaces.spider-counts.tsv: for each spider,
the features in its file, those left out for their license or for
having no usable geometry, and those in mainland China with the datum
found for them. Since the features left out aren't in
alltheplaces.parquet, a workdir without the counts is imported again
if the zip is still there.

fetch_atp also downloads AllThePlaces' own statistics for the run
(the stats_url in history.json, saved as alltheplaces.stats.json),
best effort. import_atp logs the totals of its counts, compared with
these: spiders with errors, and spiders whose number of features
differs from the dump's. On the 2026-10-31 run, the numbers of
features agree for all 5,759 spiders in the statistics; 21 spiders in
the dump aren't in them, and 872 had errors.

conflate logs how many ATP features of which spiders no matcher
handles; they get no row in conflated.parquet.

Part of #924.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttfb2LhDRExioXkdHbxLno

* fix(atp): Stop the pipeline if ATP's stats can't be fetched

Without AllThePlaces' statistics, the spider stats couldn't tell a
spider that found fewer places from one whose places we dropped, so
they are now a required input like the others: a history.json entry
without stats_url is skipped like one missing any other field, and a
failed download stops the pipeline while it is still fetching inputs.
A workdir whose sidecar predates stats_url, and that has no stats
file, gets an error that says to fetch the run again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttfb2LhDRExioXkdHbxLno

---------

Co-authored-by: Claude <noreply@anthropic.com>

Coverage Stats

419 of 427 new or added lines in 6 files covered. (98.13%)

1 existing line in 1 file now uncovered.

17475 of 18614 relevant lines covered (93.88%)

3542.96 hits per line