stacklok / toolhive / 23479389089

push

github

Retry transient network errors in background token monitor (#4281)

* Retry transient network errors in background token monitor

When a VPN disconnects overnight the OAuth token refresh endpoint becomes
temporarily unreachable. Previously the background monitor treated any
refresh failure as fatal and immediately marked the workload as
unauthenticated, requiring manual intervention.

This change distinguishes transient network errors (DNS failures,
TCP-level errors, timeouts) from real auth failures (OAuth 4xx,
invalid_grant) and retries the former with exponential backoff (30 s
initial, 5 min max, no deadline) until the network recovers. Context
cancellation stops the loop cleanly without marking the workload as
unauthenticated. Non-transient errors still fail immediately.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix unused receiver lint warnings in test helper

* Make token refresh retry intervals configurable via env vars

* Replace time.Sleep with channel synchronization in monitor tests

* Suppress false-positive gosec G101 on env var name constants

* Add per-line nosec G101 comments on env var name constants

* Test error classification through background monitor public API

* Limit OpError transient classification to syscall-level transport errors

* Inline syscall check for OpError transport classification

* Use errors.AsType consistently in isTransientNetworkError

* Move backoff retry logic from onTick into Token

* Bound token refresh retries by attempt count and elapsed time

Add WithMaxTries and WithMaxElapsedTime to the Token() retry loop so
retries are capped at 5 attempts and 5 minutes rather than running until
context cancellation or the implicit 15-minute Retry default.

Also expose both limits via environment variables and tighten the default
initial/max intervals (10s/2m instead of 30s/5m) to recover faster after
short network blips.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix integer n... (continued)

86 of 142 new or added lines in 1 file covered. (60.56%)

50858 of 79057 relevant lines covered (64.33%)

86.53 hits per line