evolution-gaming / kafka-flow / 28447066377

84%

push

Protect Kafka snapshot writes from stale writers (transactional offset binding) (#833)

* Protect Kafka snapshot writes from stale writers (transactional offset binding)

During a partition rebalance the previous and new owners can briefly overlap; a stale
flush-on-revoke can then overwrite the new owner's snapshot (issue #732). This adds an
opt-in protection for the Kafka snapshot backend.

`KafkaPersistenceModuleOf.cachingTransactional` runs snapshot writes as group-committed
Kafka transactions that also commit the input offset (`sendOffsetsToTransaction`), so the
snapshot and the offset advance atomically. Fencing is by consumer generation (KIP-447):
the broker rejects a stale generation and aborts the transaction, so neither the writes nor
the offset land. Recovery reads with `read_committed`. Output stays at-least-once.

- `Consumer` exposes `groupMetadata`, captured on assignment.
- `GroupCommit` serializes transactions per partition (lock + queue + per-item Deferred),
  binding the latest input offset; an offset-only marker forces a commit on revoke.
- `PartitionFlow` swallows a fenced offset commit on release (best-effort on revoke).
- skafka 20.1.0 -> 20.2.0 for `sendOffsetsToTransaction`/`groupMetadata`.

Tests: GroupCommitSpec (fake producer), and IT StatefulProcessingWithKafkaSpec /
TransactionalKafkaPersistenceSpec against a real broker.
Docs: docs/kafka-single-writer-design.md, persistence.md "Transactional snapshot writes".

First of three stacked PRs (kafka -> cassandra -> models).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Clarify Kafka single-writer docs and related comments

Documentation clarifications for the Kafka transactional snapshot mode (review feedback).

docs/kafka-single-writer-design.md:
- Problem diagram: show owner B committing the input offset after the snapshot
  write, so it is clear why events 101-150 are lost on the next recovery.
- Mechanism: explain that offsets are normally committ... (continued)

158 of 158 new or added lines in 7 files covered. (100.0%)

1748 of 2082 relevant lines covered (83.96%)

1.31 hits per line