evolution-gaming / kafka-journal

83%

master: 93%

LAST BUILD ON BRANCH feature/replicator-async-offset-commit

branch: feature/replicator-async-offset-commit

CHANGE BRANCH

x

Reset

• feature/replicator-async-offset-commit
• add-minor-documentation-2
• add-snapshot-projects
• binary-compatibility
• binary-compatibility1
• binary-compatibility10
• binary-compatibility11
• binary-compatibility12
• binary-compatibility13
• binary-compatibility2
• binary-compatibility3
• binary-compatibility4
• binary-compatibility5
• binary-compatibility6
• binary-compatibility7
• binary-compatibility8
• binary-compatibility9
• bincompat-smetrics-bump
• buffer-nr
• bugfix/add-missing-opts-reference-conf
• bugfix/code-visibility
• bugfix/fix-kafka-clients-version
• bugfix/fix-kafka-clients-version-for-ce2
• bugfix/replicator-async-commit-logging
• bugfix/topic-replicator-log-level
• bump-cats-helper
• bump-cats-helper-ce2
• bump-play-json-jsoniter
• cassanda-common
• cassanda-settings-deprecate-setifempty
• cassandra-sync
• ce2-remove-old-scache
• ce2/bincompat-smetrics-bump
• ce2/pointers/query-by-topic+partition
• chore/update-skafka
• chore/update-skafka-for-ce2
• consistency-config
• damper-test-fix-2
• df/DistributeJob-doc
• df/SHRIKE-73-java-kafka-producer-metrics
• df/batch-preserve-offsets
• df/correlate-events-with-meta
• df/down-crease-req-rate-to-pointer2-table
• df/expose-head-cache-config
• df/expose-set-expire-api
• df/make-data-integrity-velidation-optional
• df/pointers2
• df/reimplement-deduplication
• df/segment-and-co-docs
• df/use-kafka-topic-for-pointer
• edubrovskii/compatible-lazy-consumer-pool
• edubrovskii/kafka-consumer-pool
• extract-core
• feature/replicator-CooperativeStickyAssignor
• feature/scala-3
• feature/testcontainers-kafka
• feature/update-formatting
• fix-damper-test
• fix-insert-order
• fix-insert-order-ce2
• improve-PointerStatements
• improve-PointerStatements-1
• improve-PointerStatements-2
• isurkov/bincompat-smetrics-bump
• isurkov/ce2/bincompat-smetrics-bump
• keyspace-config
• keyspace-config-deprecation
• m/0.0.x-631
• m/4.0.0
• m/636-expose-new-temporary-metric(s)-on-when-pointer-table-is-used-as-fallback
• m/638-drop-`pointer`-table
• m/812-remove-reading-from-and-writing-to-`pointer`-table
• m/add-explicit-types-everywhere
• m/add-pekko-module
• m/batch-investigation
• m/batch-investigation-reworked-logic
• m/cats-effect-3.6.3
• m/change-package
• m/collect-journal-expiration-values-as-metric
• m/expose-client-version-as-metric
• m/fix-unapplied-migrations
• m/improve-local-build
• m/introduce-usage-of-release-workflow
• m/make-version-non-optional
• m/notes-for-`Next-Developer`
• m/pre-676
• m/prepare-for-5.0.0
• m/prepare-for-pekko-support
• m/remove-deprecated-code
• m/remove-the-fallbacks-and-release-the-version-#637
• m/scalafmt
• m/update-building-infrastructure
• m/update-dependencies
• m/update-scassandra-to-5.2.1
• m/upgrade-warnings-to-errors
• m/use-Oracle-JDK-for-build
• m/use-nicer-package-for-pekko-artifacts
• m/version
• master
• migesok/replicated-offset-notifier
• migrate-schema
• more-scaladocs
• more-scaladocs-10
• more-scaladocs-11
• more-scaladocs-12
• more-scaladocs-13
• more-scaladocs-14
• more-scaladocs-15
• more-scaladocs-16
• more-scaladocs-17
• more-scaladocs-18
• more-scaladocs-19
• more-scaladocs-2
• more-scaladocs-20
• more-scaladocs-21
• more-scaladocs-22
• more-scaladocs-23
• more-scaladocs-24
• more-scaladocs-4
• more-scaladocs-5
• more-scaladocs-6
• more-scaladocs-7
• more-scaladocs-8
• more-scaladocs-9
• pointers/always-query-by-topic-and-partition
• pointers/drop-select-all-partitions-api
• re-implement_purge_delete
• refs/tags/v0.3.13
• refs/tags/v0.4.0
• remove-old-scache
• revert-547-update/smetrics-2.2.0
• row-was-applied
• scalafix-config
• series/0.0.x
• settings-config
• simplify-create-schema
• snapshot
• snapshot-class
• snapshot-record
• snapshot-store
• split-schema-creation
• testcontainers-cassandra
• testcontainers-cassandra-ce2
• update-akka-serialization-docs
• update-sbt
• update-scache
• update-scache-ce2
• update-scala
• update-scassandra
• update/akka-test-actor-0.3.0
• update/cassandra-1.21.0
• update/cassandra-1.21.1
• update/cassandra-1.21.2
• update/cassandra-1.21.3
• update/cats-core-2.13.0
• update/cats-helper-3.12.0
• update/cats-helper-3.12.2
• update/circe-core-0.14.12
• update/circe-core-0.14.13
• update/circe-core-0.14.14
• update/circe-core-0.14.15
• update/executor-tools-1.0.5
• update/hostname-0.3.0
• update/kind-projector-0.13.4
• update/log4j-over-slf4j-2.0.17
• update/logback-classic-1.5.13
• update/logback-classic-1.5.14
• update/logback-classic-1.5.15
• update/logback-classic-1.5.16
• update/logback-classic-1.5.17
• update/logback-classic-1.5.18
• update/logback-classic-1.5.19
• update/logback-classic-1.5.20
• update/pekko-extension-test-actor-0.0.4
• update/pekko-persistence-1.1.5
• update/pekko-persistence-1.2.0
• update/pekko-persistence-1.2.1
• update/play-json-2.10.7
• update/play-json-2.10.8
• update/play-json-3.0.5
• update/play-json-3.0.6
• update/play-json-jsoniter-1.2.3
• update/pureconfig-core-0.17.9
• update/sbt-1.10.10
• update/sbt-1.10.11
• update/sbt-1.10.7
• update/sbt-1.10.8
• update/sbt-1.11.0
• update/sbt-1.11.1
• update/sbt-1.11.3
• update/sbt-1.11.4
• update/sbt-1.11.5
• update/sbt-1.11.6
• update/sbt-1.11.7
• update/sbt-coveralls-1.3.15
• update/sbt-dynver-5.1.1
• update/sbt-jmh-0.4.8
• update/sbt-scalafix-0.14.0
• update/sbt-scalafix-0.14.2
• update/sbt-scalafmt-2.5.3
• update/sbt-scalafmt-2.5.4
• update/sbt-scalafmt-2.5.6
• update/sbt-scoverage-2.3.0
• update/sbt-scoverage-2.3.1
• update/sbt-scoverage-2.4.0
• update/scache-5.1.3
• update/scala3-library-3.3.6
• update/scala3-library-3.3.7
• update/scalafmt-core-3.10.0
• update/scalafmt-core-3.10.1
• update/scalafmt-core-3.8.4
• update/scalafmt-core-3.8.5
• update/scalafmt-core-3.8.6
• update/scalafmt-core-3.9.0
• update/scalafmt-core-3.9.1
• update/scalafmt-core-3.9.10
• update/scalafmt-core-3.9.2
• update/scalafmt-core-3.9.3
• update/scalafmt-core-3.9.5
• update/scalafmt-core-3.9.6
• update/scalafmt-core-3.9.7
• update/scalafmt-core-3.9.8
• update/scalafmt-core-3.9.9
• update/scodec-bits-1.2.1
• update/scodec-bits-1.2.2
• update/scodec-bits-1.2.3
• update/scodec-bits-1.2.4
• update/scodec-core-1.11.11
• update/scodec-core-2.3.3
• update/skafka-17.1.4
• update/skafka-17.2.0
• update/skafka-17.2.1
• update/skafka-17.2.2
• update/smetrics-2.3.0
• update/smetrics-2.3.2
• update/smetrics-2.3.4
• update/testcontainers-scala-cassandra-0.41.5
• update/testcontainers-scala-cassandra-0.41.8
• update/testcontainers-scala-cassandra-0.43.0
• v3.2.1
• v3.2.2
• v3.2.3
• v3.2.4
• v3.3.0
• v3.3.1
• v3.3.2
• v3.3.3
• v3.3.4
• v3.3.5
• v3.3.6
• v3.3.7
• v3.3.8
• v3.3.9
• v3.4.0
• v3.4.1
• v3.4.2
• v3.5.0
• v3.5.1
• v3.6.0
• v4.0.0
• v4.0.1
• v4.0.2
• v4.1.0
• v4.1.1
• v4.1.2
• v4.1.3
• v4.1.4
• v4.1.5
• v4.1.6
• v4.1.7
• v4.1.8
• v4.2.0
• v4.2.1
• v4.3.0
• v4.3.1
• v5.0.0
• v5.0.0-RC1
• v5.0.0-RC2
• v5.0.0-RC3
• v5.0.0-RC4
• v5.0.0-RC5
• v5.1.0
• v5.2.0
• v5.2.1
• wip/scala-3
• yklymko/kafka-consumer-pool

Build # 14332360101

Build Type

Pull #739

Committed by migesok

Commit Message

New improved Kafka offset commit logic for replicator

The next release version should have a minor increment - 4.2.0

- TopicCommit.apply renamed to TopicCommit.sync, TopicCommit.apply deprecated
- Added TopicCommit.asyncPeriodic implementation
- TopicCommit.delayed deprecated in favor of TopicCommit.asyncPeriodic
- TopicCommit.asyncPeriodic is used by default in the replicator
- updated skafka for an async commit method fix

Main features of asyncPeriodic:
- periodic commits are done asynchronously without blocking the main poll loop, improving latency
- it properly handles rebalance, not loosing progress for revoked partitions
- same logic commits offsets on app shutdown because Java consumer invokes revoke partition callbacks
  on closure
- gracefully handles RebalanceInProgressException

Additionally:
- made TopicCommit journal package private - it is used only in journal package private APIs,
  and it is not supposed to be used outside the internal journal code
- ToTry[StateT] impl in ConsumeTopicTest - it didn't work on flatMap'ed RebalanceCallback's

Pull Request Pull Request #739: New improved Kafka offset commit logic for replicator

Run Details

61 of 78 new or added lines in 5 files covered. (78.21%)

16 existing lines in 13 files now uncovered.

3224 of 3885 relevant lines covered (82.99%)

1.41 hits per line