tari-project / tari / 38393174892

77%

Build Type

push

github

Committed by web-flow

Commit Message

fix(comms): bound memory held for stalled yamux peers, RPC/sync streams and gRPC GetBlocks (#8108)

## Description

This PR puts hard per-connection and per-session limits on the memory
the comms stack and the node's gRPC `GetBlocks` can hold on behalf of a
remote peer or client. A mainnet archive node on 6.1.0 was OOM-killed
after about 12 h. These changes remove the worst cases whether or not
they turn out to be that OOM's cause:

- `sync_blocks` serving read-ahead: about 154 MiB per session (about 15
GiB at the 100-session cap).
- RPC client stream buffers: up to about 201 MiB per streaming request.
- The yamux pin: about 128 MiB per stalled connection, held
indefinitely.
- `GetBlocks`: unbounded work and memory for abandoned or non-reading
calls.

There are three commits, one per area. They apply independently: the RPC
and gRPC commits don't depend on the yamux commit.

### 1. `fix(comms)`: bound the memory a stalled yamux peer can pin

- **Write-stall watchdog.** The raw transport, under noise, is wrapped
(`WriteStallSocket`) to record when writes start waiting. The clock
restarts whenever the transport accepts any bytes, including partial
writes of an encrypted frame, so slow but working links are never
flagged. yamux reuses the same `WriteStall`
(`Yamux::upgrade_connection_with_write_stall`).
  - A connection whose writes have waited 30 s is reported as stalled.
- At 120 s it is closed with a WARN naming the peer and the stall
duration. There is no ban.
- This is the only thing that frees a connection yamux can't clean up:
yamux only drops dead streams when its write path is idle.
- **Stream cap 512 → 128.** yamux 0.13 closes the whole connection when
an open would go past the limit, in either direction. The worker
therefore counts its streams, including one being handed over, and
refuses our own opens at 112 (16 headroom) with `TooManyStreams`. The
connection stays up.
- A locally refused open is not the peer's fault, so header and horizon
syn... (continued)

Coverage Stats

1040 of 1105 new or added lines in 18 files covered. (94.12%)

515 existing lines in 30 files now uncovered.

125683 of 168289 relevant lines covered (74.68%)

446142.77 hits per line