evolution-gaming / kafka-flow / 29927664876

83%

Build Type

push

Committed by web-flow

Commit Message

Recover transactional snapshots completely, with a bounded read (#857)

* Recover transactional snapshots completely, with a bounded read

The recovery read stopped at the read consumer's own end offset -
under read_committed that is the last-stable-offset, which an open
transaction pins below records committed after it. A recovery inside
that window silently missed committed snapshots while input offsets
were not held back the same way, so a second handover could recover
stale state yet resume from the newer committed offset. The read now
targets the high watermark, captured up front through a short-lived
read_uncommitted consumer, and waits out any open transaction below
it - the shape Kafka Streams' exactly-once restore settled on
(KAFKA-10167). A stable per-partition transactional.id makes the wait
rare: a takeover's initTransactions aborts a crashed predecessor's
unfinished transaction sub-second. Fencing of stale writers stays
with the consumer generation, so completeness rests on the read
bound, never on naming.

That wait is normally self-resolving, but a hanging transaction
(KIP-664; prevented by KIP-890 brokers) or a log truncated below the
captured target made the read hang forever on the poll thread inside
the rebalance callback, until max.poll.interval.ms evicted the member
silently. A no-progress deadline (default 3 minutes, reset by
every position advance) fails the read with
RecoveryReadStalledError instead, diagnosed against a re-read log
end - truncation versus a transaction that outlived the deadline -
freeing the thread for a restarted recovery.

The mode's public surface stays the entry point plus
TransactionalConfig: the stall error is package-private (its contract
is the log line and its diagnosis, not a catchable type), and both
knob defaults live on a TransactionalConfig companion as
package-private constants - the config owns its defaults, the
mechanisms take explicit parameters. Narrowing the previously public
group-commit d... (continued)

Coverage Stats

30 of 31 new or added lines in 2 files covered. (96.77%)

110 existing lines in 33 files now uncovered.

1821 of 2181 relevant lines covered (83.49%)

1.33 hits per line