uber / h3-go / 38609426000

100%

push

github

fix: take latitude from atan2 near the poles, in C and in x/h3go (#167)

vec3ToLatLng computed latitude as asin(z). Near the poles asin recovers
cos(lat) from 1 - z², which cancels, so the latitude error grows as
1/cos(lat): the res 4 cell at 89.96 degrees north came out 6.9e-12
degrees of arc from the exact point, about 540 ulps, while the unit
vector it was converted from was within four. The pure-Go port had the
same formula and the same loss, and because the two losses are
independent the port and the reference disagreed by 1e-10 degrees at
polar cells from res 7 and by 1.8e-7 at res 15, far over the suite's
1e-11 tolerance; the suite passed only because its sampler never draws
those cells.

Change the latitude to atan2(z, sqrt(x² + y²)), which is the same
function on the unit sphere but takes cos(lat) from x and y, where it is
held at full precision. The vendored C gets the upstream change ahead
of its release, with its comment, so the files and the port move
together; the next vendoring run replaces it with the released source.
x/h3go gets the same line. Measured against the regenerated files, the
worst separation between port and reference is 6.7e-14 degrees at every
latitude and resolution, including a polar beam search to res 15, down
from 1.6e-12 over res 0 to 4 and 1.8e-7 at polar res 15; the worst
sampled cases now sit near the equator. Mixing the two formulas is the
worst configuration, which is why both change in one commit.

Regenerate the suite from the patched library. Only the float and
region files change, in the last bits of points and polygon inputs; no
cell set and no digest row changes, so the resolution 7 digest rows
stand. Tighten the manifest's angularDeg from 1e-11 to 1e-12, which
leaves fifteen times headroom over the measured worst, and the shared
point bound of the invariants to match.

One root package test held the square-metre area of a cell to four
decimal places; the vertices moving by last bits moved that area by
1.... (continued)

1 of 1 new or added line in 1 file covered. (100.0%)

4550 of 4550 relevant lines covered (100.0%)

4166462.97 hits per line