aio-libs / multidict / 36130559673

87%

push

github

Bound the table size a length hint can reserve (#1577)

<!-- Thank you for your contribution! -->

## What do these changes do?

Hardens the C extension's size arithmetic against an object whose
`__length_hint__()` reports close to `sys.maxsize`.

`md_reserve_for_upd()` computed `(extra_size + used) * 3 + 1` without an
overflow check, and `_md_resize()` only rejected sizes of 64 bits or
more. A size of 2**61 to 2**63 still passed, and `_htkeys_alloc_size()`
then shifted `1` by 64 or 65. Under `MULTIDICT_ASAN_BUILD=1` that aborts
with `shift exponent 65 is too large for 64-bit type`; on a release
build it happened to end in `MemoryError`. Separately,
`_multidict_extend_parse_args()` added the keyword count to the hint,
wrapped negative, and `extend()`/`update()`/`merge()` returned NULL with
no exception set (`SystemError`).

The fix saturates the keyword addition, ignores a hint too large to
estimate for (as `list.extend()` does when size plus hint overflows),
and adds `htkeys_size_fits()` so `_md_resize()` rejects any size whose
byte count does not fit in `Py_ssize_t`, before any shift happens. The
pure-Python backend does no such arithmetic and is unchanged.

## Are there changes in behavior for the user?

A `__length_hint__()` too large to estimate for is ignored instead of
raising `MemoryError`, and passing keyword arguments alongside it no
longer raises `SystemError`.

## Is it a substantial burden for the maintainers to support this?

No. One comparison on the reserve path and a helper on the resize path,
which is already cold.

## Related issue number

N/A

## Checklist

- [x] I think the code is well written
- [x] Unit tests for the changes exist
- [x] Documentation reflects the changes (N/A, no documented behaviour
changed)
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt` (N/A)
- [x] Add a new news fragment into the `CHANGES/` folder
- [x] `make doc-spelling` passes and any new technical words are added
to `doc... (continued)

779 of 1558 branches covered (50.0%)

Branch coverage included in aggregate %.

24 of 27 new or added lines in 1 file covered. (88.89%)

8007 of 8536 relevant lines covered (93.8%)

1.88 hits per line