c-ares / c-ares / 28789997333

90%

push

github

Fix sticky server recovery when all servers have failures (#1192)

## Problem

In the review discussion on #992, @DavidMcNallyAlianza and @Flynnzaa
identified a failure mode they named **"sticky recovery"**: when every
configured server is failing, a server that recovers from an extended
outage can go unnoticed for an unbounded amount of time. Credit to them
for the analysis and the timeline diagram that made this concrete.

Their scenario (2 servers, 1 qps): both servers fail for 10 minutes
(both counters reach 600). Server A briefly recovers — resetting its
counter to 0 — then starts failing again, while server B's counter stays
frozen at 600 because no traffic reaches it. When B actually recovers
moments later, nothing notices: all queries keep going to A (count ~60
and climbing) until A's counter climbs past B's frozen 600. That's ~9
minutes of avoidable downtime in their example, and in general the
blindness window scales with the length of the original outage, at any
query rate.

Two root causes in the current code:

1. **Probing is gated off exactly when it's needed most.**
`ares_send_query()` only considers spawning a probe when the triggering
query was sent to a server with *no* failures. During a total outage
every query necessarily goes to a failed server, so no probes are ever
sent, and the "auto-rotate on each attempt" property of the sorted
server list only holds while failure counts stay within ±1 of each other
— a recovery blip breaks that symmetry permanently.
2. **The consecutive-failure count is unbounded and freezes when traffic
stops.** Only the *relative order* of the counts is used for selection,
so magnitude beyond "clearly down" carries no signal — but a huge frozen
count means an equally huge number of failures elsewhere before the
server is ever consulted again.

## Fix (three small changes, no new state, no new options)

1. **Relax the probe trigger** (`ares_process.c`): a fresh query sent to
a failed server can now spawn a... (continued)

114 of 114 new or added lines in 3 files covered. (100.0%)

24253 of 26967 relevant lines covered (89.94%)

16306.33 hits per line