tari-project / tari / 33423629872

77%

Build Type

push

github

Committed by web-flow

Commit Message

fix: stop running peer-database SQLite on tokio worker threads (#7981)

Every `PeerManager` method was an `async fn` with no await points that
called straight into diesel/r2d2. That blocks the *worker thread*, not
just the task, so a contended peer database drained the tokio worker
pool one thread at a time until the comms actors stopped being
scheduled: the node kept its TCP connections but could no longer dial or
propagate, and only a restart recovered it. The error that surfaced was
four layers away, in the outbound pipeline, reporting a deadlock in a
pipeline that was nearly empty.

Root cause fix
- Add `BlockingPeerStorage`, which owns the only `PeerStorageSql` handle
and exposes it solely through `call()`, dispatching to
`tokio::task::spawn_blocking`. The handle is private and has no
accessor, so a synchronous peer database call from `PeerManager` no
longer compiles.

Supporting changes
- Scope the peer database's `PRAGMA busy_timeout` to 10s
(`PEER_DATABASE_BUSY_TIMEOUT`) via new
`DbConnection::connect_{url,and_migrate}_with_busy_timeout`. The shared
60s default is unchanged for every other database, including the
wallet's, where losing a write is worse than waiting.
- Shed slow peer lookups in actor context: the ban check in
`ConnectivityManager::handle_dial_peer`, the lookup in
`ConnectionManager::dial_peer` and the dialer's peer write-back are now
bounded by `PEER_LOOKUP_TIMEOUT` (5s). The query still completes on the
blocking pool; the actor just stops waiting for it.
- Back off unresolvable dials in the DHT connectivity pool. Failed dials
now carry per-peer exponential backoff (30s doubling to 10m), so a pool
that never becomes healthy no longer re-dials the same unreachable peers
every cycle - which was the write load that tipped the peer database
over, since every failed dial writes back through `add_or_update_peer`.
- Make the outbound pipeline's timeout message distinguish "pipeline
saturated" from "downstream actor unresponsive", usin... (continued)

Coverage Stats

1300 of 1485 new or added lines in 16 files covered. (87.54%)

212 existing lines in 29 files now uncovered.

78728 of 123516 relevant lines covered (63.74%)

338369.48 hits per line