diffed-places / pipeline / 23642542246

93%

Build Type

push

github

Committed by web-flow

Commit Message

Compute bitmasks for feature matching, increase matching distance (#217)

We now use a maximal matching distance of 400 meters for shops, 800 meters
for railway platforms, and 10 meters for “tiny” features such as hydrants,
trees or trash cans.

We now partition both ATP and OSM features into “matching classes”
such as `SHOP` or `STREET_FURNITURE`. A single feature may belong
to multiple classes. This is not intended as yet another POI category
system; we will only use it to restrict the input to actual feature
matching.

The pipeline now filters out any features that don’t belong to any
matching class, thus excluding streets or buildings that aren’t of
interest to the diffed places project.

Tmings on a 2018 MacBook Pro with 6 Intel CPUs after this change:

| Phase            | Walltime |
|------------------|---------:|
| atp.fetch        |     76 s |
| atp.read         |     51 s |
| atp.write        |    186 s |
| coverage.read    |   1101 s |
| coverage.write   |   1337 s |
| osm.fetch        |        — |
| osm.prt.r        |      1 s |
| osm.cov.n        |    741 s |
| osm.cov.w        |    261 s |
| osm.cov.r        |      6 s |
| osm.filter.r     |     57 s |
| osm.filter.w     |   3308 s |
| osm.filter.n     |   5846 s |
| osm.assemble.r   |   1403 s |
| osm.assemble.w   |    842 s |

The generated Parquet file is now 293 MiB for AllThePlaces,
and 938 MiB for OpenStreetMap.

Coverage Stats

185 of 197 new or added lines in 6 files covered. (93.91%)

12 existing lines in 3 files now uncovered.

2783 of 2978 relevant lines covered (93.45%)

261.78 hits per line