aio-libs / yarl / 36573901174

98%

push

github

Percent-decode special-scheme hosts in WHATWG mode (#1942)

<!-- Thank you for your contribution! -->

## What do these changes do?

In the default WHATWG mode, the hosts of the `http`, `https`, `ws`,
`wss`, `ftp` and `file` schemes are now percent-decoded, as the WHATWG
host parser does. The decoded bytes are read as UTF-8, and the result
goes through the same IDNA encoding and IPv4 parsing as any other host.
So `http://ex%41mple.com/` becomes `http://example.com/`,
`https://%e2%98%83/` becomes `https://xn--n3h/` and `http://%30x7f.1/`
becomes `http://127.0.0.1/`. A host that decodes to something invalid,
such as `ho%00st`, `a%2Fb` or invalid UTF-8, raises `ValueError`. The
same applies to `URL.build()`, `with_host()`, `with_scheme()`, `join()`
and switching mode. RFC 3986 mode, non-special schemes and
`encoded=True` keep the host as written.

This moves 67 rows of the conformance report into "yarl follows each
standard in its mode" (99 to 166). The "yarl differs, RFC 3986 and
WHATWG agree" section is unchanged (only `http://[::%31]`). Most of the
remaining 7 rows in this category are hosts with a default-ignorable
code point such as U+00AD, which yarl rejects on purpose.

## Are there changes in behavior for the user?

Yes. In WHATWG mode, a percent-encoded host of a special scheme is
decoded, or rejected when the decoded host is invalid. Before, it was
kept as written.

## Is it a substantial burden for the maintainers to support this?

No. The decoding lives in one helper next to the IPv4 parser from #1940
and is cached with the host encoding. Hosts without `%` take the same
path as before.

## Related issue number

Follows #1940.

## Checklist

- [x] I think the code is well written
- [x] Unit tests for the changes exist
- [x] Documentation reflects the changes
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt`
- [x] Add a new news fragment into the `CHANGES/` folder

Drafted with Claude Code (Claude Opus 5.5); revi... (continued)

97 of 194 branches covered (50.0%)

Branch coverage included in aggregate %.

144 of 144 new or added lines in 7 files covered. (100.0%)

9184 of 9308 relevant lines covered (98.67%)

0.99 hits per line