aio-libs / yarl / 36773169349

98%

push

github

Reject characters outside the grammar in RFC 3986 mode (#1953)

<!-- Thank you for your contribution! -->

## What do these changes do?

In RFC 3986 mode, parsing a URL string now rejects characters outside
the grammar of RFC 3986 and RFC 3987 with `ValueError`, instead of
percent-encoding them. This follows the rejection of `\` in RFC mode
from #1943, and uses RFC 3986+3987 as the baseline, as the conformance
oracle does since #1949.

Rejected: a space, `"`, `<`, `>`, `^`, a backtick, `{`, `|`, `}`,
control characters, `[` or `]` outside an IP-literal host, characters
inside the brackets outside IPv6address, IPvFuture and the RFC 6874 zone
identifier (a space, a bare `%`, non-ASCII), a second `#`, a `%` that
does not start a percent-encoded octet, a non-digit in the port, code
points that RFC 3987 excludes (C1 controls, noncharacters such as
U+FFFF, bidi formatting characters, private use outside the query), an
invalid scheme, and a `:` in the first segment of a relative path (RFC
3986 section 4.2). Leading and trailing spaces and controls are no
longer stripped in this mode, nor are tabs and newlines removed.
Non-ASCII characters allowed in an IRI are still percent-encoded, and
non-ASCII hosts are still encoded with IDNA2008.

The check runs only in the RFC mode entry point (`encode_url_rfc`), so
WHATWG mode does not pay for it; the RFC-only `_check_rfc_authority()`
call and the RFC branch of `_backslashes()` move into it. Built
components (`URL.build()`, `with_path()`, `/`, `with_query()` and so on)
and `encoded=True` are unchanged, except that RFC 3986 mode now encodes
every `:` in the first segment of a relative path as `%3A` (section
4.2), so that `str(URL.build(path=":foo", mode="rfc"))` parses back;
WHATWG mode keeps encoding only a scheme-shaped one.

Report rows against master:

| From | To | Rows |
|---|---|---|
| RFC mode differs from RFC 3986+3987 | yarl follows each standard in
its mode | 60 |
| Both modes differ from their standard | WH... (continued)

97 of 194 branches covered (50.0%)

Branch coverage included in aggregate %.

189 of 189 new or added lines in 4 files covered. (100.0%)

9527 of 9654 relevant lines covered (98.68%)

0.99 hits per line