aio-libs / yarl / 36719417094

98%

push

github

Accept IRIs in the RFC 3986 oracle of the conformance report (#1949)

<!-- Thank you for your contribution! -->

## What do these changes do?

The RFC 3986 oracle in `tools/conformance/rfc3986_oracle.py` now accepts
IRIs and maps them to URIs as RFC 3987 section 3.1 does, before
resolving and comparing. A non-ASCII host goes through the RFC 5895
mapping and IDNA2008 when it is a valid IDNA2008 name and is
percent-encoded otherwise, as section 3.1 makes ToASCII optional; other
non-ASCII characters are percent-encoded as UTF-8. `compare.py` checks a
table of oracle answers for inputs the corpus does not cover on every
run. Code points outside the RFC 3987 grammar (C1 controls,
noncharacters such as U+FFFF, private-use characters outside the query)
and the bidi formatting characters forbidden by RFC 3987 section 4.1
stay invalid.

RFC 5895 is used rather than UTS #46 because UTS #46 silently deletes
invisible code points such as U+200B and U+00AD in hosts, which yarl
deliberately rejects.

This prepares a follow-up that makes RFC 3986 mode reject characters
outside the grammar instead of percent-encoding them, with RFC 3987 as
the baseline for non-ASCII.

Row movements in `REPORT.md` against master; "yarl differs, RFC 3986 and
WHATWG agree" still has only `http://[::%31]`:

| From | To | Rows |
|---|---|---|
| RFC mode differs from RFC 3986 | RFC 3986, WHATWG and yarl in both
modes agree | 18 |
| RFC mode differs from RFC 3986 | yarl follows each standard in its
mode | 1 |
| Both modes differ from their standard | WHATWG mode differs from
WHATWG, RFC mode follows RFC 3986 | 4 |
| RFC 3986, WHATWG and yarl in both modes agree | RFC mode differs from
RFC 3986 | 6 |
| WHATWG mode differs from WHATWG, RFC mode follows RFC 3986 | Both
modes differ from their standard | 2 |

The last two rows are hosts that are not valid IDNA2008 names
(`http://💩.123/`, `https://\xad/`, `http://％４１.com`,
`http://GOO\u200bgoo.com`, ...). RFC 3987 percent-encodes them; RFC 3986
... (continued)

97 of 194 branches covered (50.0%)

Branch coverage included in aggregate %.

9361 of 9488 relevant lines covered (98.66%)

0.99 hits per line