evolution-gaming / kafka-journal

83%

master: 94%

LAST BUILD ON BRANCH v4.1.7

branch: v4.1.7

CHANGE BRANCH

x

Reset

• v4.1.7
• add-minor-documentation-2
• add-snapshot-projects
• binary-compatibility
• binary-compatibility1
• binary-compatibility10
• binary-compatibility11
• binary-compatibility12
• binary-compatibility13
• binary-compatibility2
• binary-compatibility3
• binary-compatibility4
• binary-compatibility5
• binary-compatibility6
• binary-compatibility7
• binary-compatibility8
• binary-compatibility9
• bincompat-smetrics-bump
• buffer-nr
• bugfix/add-missing-opts-reference-conf
• bugfix/code-visibility
• bugfix/fix-kafka-clients-version
• bugfix/fix-kafka-clients-version-for-ce2
• bugfix/replicator-async-commit-logging
• bugfix/topic-replicator-log-level
• bump-cats-helper
• bump-cats-helper-ce2
• bump-play-json-jsoniter
• cassanda-common
• cassanda-settings-deprecate-setifempty
• cassandra-sync
• ce2-remove-old-scache
• ce2/bincompat-smetrics-bump
• ce2/pointers/query-by-topic+partition
• chore/update-skafka
• chore/update-skafka-for-ce2
• consistency-config
• damper-test-fix-2
• df/DistributeJob-doc
• df/SHRIKE-73-java-kafka-producer-metrics
• df/batch-preserve-offsets
• df/correlate-events-with-meta
• df/down-crease-req-rate-to-pointer2-table
• df/expose-head-cache-config
• df/expose-set-expire-api
• df/make-data-integrity-velidation-optional
• df/pointers2
• df/reimplement-deduplication
• df/segment-and-co-docs
• df/use-kafka-topic-for-pointer
• edubrovskii/compatible-lazy-consumer-pool
• edubrovskii/kafka-consumer-pool
• extract-core
• feature/replicator-CooperativeStickyAssignor
• feature/replicator-async-offset-commit
• feature/scala-3
• feature/testcontainers-kafka
• feature/update-formatting
• fix-damper-test
• fix-insert-order
• fix-insert-order-ce2
• improve-PointerStatements
• improve-PointerStatements-1
• improve-PointerStatements-2
• isurkov/bincompat-smetrics-bump
• isurkov/ce2/bincompat-smetrics-bump
• keyspace-config
• keyspace-config-deprecation
• m/0.0.x-631
• m/4.0.0
• m/636-expose-new-temporary-metric(s)-on-when-pointer-table-is-used-as-fallback
• m/add-explicit-types-everywhere
• m/add-pekko-module
• m/batch-investigation
• m/batch-investigation-reworked-logic
• m/cats-effect-3.6.3
• m/change-package
• m/collect-journal-expiration-values-as-metric
• m/expose-client-version-as-metric
• m/improve-local-build
• m/introduce-usage-of-release-workflow
• m/make-version-non-optional
• m/notes-for-`Next-Developer`
• m/pre-676
• m/prepare-for-5.0.0
• m/prepare-for-pekko-support
• m/remove-deprecated-code
• m/remove-the-fallbacks-and-release-the-version-#637
• m/scalafmt
• m/update-building-infrastructure
• m/update-dependencies
• m/update-scassandra-to-5.2.1
• m/upgrade-warnings-to-errors
• m/use-Oracle-JDK-for-build
• m/use-nicer-package-for-pekko-artifacts
• m/version
• master
• migesok/replicated-offset-notifier
• migrate-schema
• more-scaladocs
• more-scaladocs-10
• more-scaladocs-11
• more-scaladocs-12
• more-scaladocs-13
• more-scaladocs-14
• more-scaladocs-15
• more-scaladocs-16
• more-scaladocs-17
• more-scaladocs-18
• more-scaladocs-19
• more-scaladocs-2
• more-scaladocs-20
• more-scaladocs-21
• more-scaladocs-22
• more-scaladocs-23
• more-scaladocs-24
• more-scaladocs-4
• more-scaladocs-5
• more-scaladocs-6
• more-scaladocs-7
• more-scaladocs-8
• more-scaladocs-9
• pointers/always-query-by-topic-and-partition
• pointers/drop-select-all-partitions-api
• re-implement_purge_delete
• refs/tags/v0.3.13
• refs/tags/v0.4.0
• remove-old-scache
• revert-547-update/smetrics-2.2.0
• row-was-applied
• scalafix-config
• series/0.0.x
• settings-config
• simplify-create-schema
• snapshot
• snapshot-class
• snapshot-record
• snapshot-store
• split-schema-creation
• testcontainers-cassandra
• testcontainers-cassandra-ce2
• update-akka-serialization-docs
• update-sbt
• update-scache
• update-scache-ce2
• update-scala
• update-scassandra
• update/akka-test-actor-0.3.0
• update/cassandra-1.21.0
• update/cassandra-1.21.1
• update/cassandra-1.21.2
• update/cassandra-1.21.3
• update/cats-core-2.13.0
• update/cats-helper-3.12.0
• update/cats-helper-3.12.2
• update/circe-core-0.14.12
• update/circe-core-0.14.13
• update/circe-core-0.14.14
• update/executor-tools-1.0.5
• update/hostname-0.3.0
• update/log4j-over-slf4j-2.0.17
• update/logback-classic-1.5.13
• update/logback-classic-1.5.14
• update/logback-classic-1.5.15
• update/logback-classic-1.5.16
• update/logback-classic-1.5.17
• update/logback-classic-1.5.18
• update/pekko-extension-test-actor-0.0.4
• update/pekko-persistence-1.1.5
• update/pekko-persistence-1.2.0
• update/play-json-2.10.7
• update/play-json-3.0.5
• update/play-json-jsoniter-1.2.3
• update/pureconfig-core-0.17.9
• update/sbt-1.10.10
• update/sbt-1.10.11
• update/sbt-1.10.7
• update/sbt-1.10.8
• update/sbt-1.11.0
• update/sbt-1.11.1
• update/sbt-1.11.3
• update/sbt-1.11.4
• update/sbt-1.11.5
• update/sbt-1.11.6
• update/sbt-coveralls-1.3.15
• update/sbt-dynver-5.1.1
• update/sbt-scalafix-0.14.0
• update/sbt-scalafix-0.14.2
• update/sbt-scalafmt-2.5.3
• update/sbt-scalafmt-2.5.4
• update/sbt-scoverage-2.3.0
• update/sbt-scoverage-2.3.1
• update/scache-5.1.3
• update/scala3-library-3.3.6
• update/scalafmt-core-3.8.4
• update/scalafmt-core-3.8.5
• update/scalafmt-core-3.8.6
• update/scalafmt-core-3.9.0
• update/scalafmt-core-3.9.1
• update/scalafmt-core-3.9.10
• update/scalafmt-core-3.9.2
• update/scalafmt-core-3.9.3
• update/scalafmt-core-3.9.5
• update/scalafmt-core-3.9.6
• update/scalafmt-core-3.9.7
• update/scalafmt-core-3.9.8
• update/scalafmt-core-3.9.9
• update/scodec-bits-1.2.1
• update/scodec-bits-1.2.2
• update/scodec-bits-1.2.3
• update/scodec-bits-1.2.4
• update/scodec-core-1.11.11
• update/scodec-core-2.3.3
• update/skafka-17.1.4
• update/skafka-17.2.0
• update/skafka-17.2.1
• update/skafka-17.2.2
• update/smetrics-2.3.0
• update/smetrics-2.3.2
• update/testcontainers-scala-cassandra-0.41.5
• update/testcontainers-scala-cassandra-0.41.8
• update/testcontainers-scala-cassandra-0.43.0
• v3.2.1
• v3.2.2
• v3.2.3
• v3.2.4
• v3.3.0
• v3.3.1
• v3.3.2
• v3.3.3
• v3.3.4
• v3.3.5
• v3.3.6
• v3.3.7
• v3.3.8
• v3.3.9
• v3.4.0
• v3.4.1
• v3.4.2
• v3.5.0
• v3.5.1
• v3.6.0
• v4.0.0
• v4.0.1
• v4.0.2
• v4.1.0
• v4.1.1
• v4.1.2
• v4.1.3
• v4.1.4
• v4.1.5
• v4.1.6
• v4.1.8
• v4.2.0
• v4.2.1
• v4.3.0
• v4.3.1
• v5.0.0
• v5.0.0-RC1
• v5.0.0-RC2
• v5.0.0-RC3
• v5.0.0-RC4
• v5.0.0-RC5
• wip/scala-3
• yklymko/kafka-consumer-pool

Build # 13242495144

Build Type

push

Committed by web-flow

Commit Message

Add ReplicatedOffsetNotifier callback (#720)

Current recovery logic doesn't scale when the volume of events is large,
because clients have to consume all the events on all the app nodes.
Additionally, current logic creates Kafka consumers on-demand, which is
a heavy operation. This consumer creation happens in spikes, when
replicator is running late, and affects client applications stability.

Introducing a side channel for notifications about replication progress
will allow us in the future to have an alternative implementation for
recovery without these flaws. I.e. notifications go to a separate lean
Kafka topic which is read by all client nodes. On recovery:
- a mark is written
- wait until mark offset confirmed replicated in the notification topic
(with potential fallback to polling of the pointer table in Cassandra)
- recover from Cassandra only

Run Details

6 of 7 new or added lines in 3 files covered. (85.71%)

14 existing lines in 12 files now uncovered.

3164 of 3816 relevant lines covered (82.91%)

1.38 hits per line