evolution-gaming / kafka-journal

83%

master: 94%

LAST BUILD ON BRANCH migesok/replicated-offset-notifier

branch: migesok/replicated-offset-notifier

CHANGE BRANCH

x

Reset

• migesok/replicated-offset-notifier
• add-minor-documentation-2
• add-snapshot-projects
• binary-compatibility
• binary-compatibility1
• binary-compatibility10
• binary-compatibility11
• binary-compatibility12
• binary-compatibility13
• binary-compatibility2
• binary-compatibility3
• binary-compatibility4
• binary-compatibility5
• binary-compatibility6
• binary-compatibility7
• binary-compatibility8
• binary-compatibility9
• bincompat-smetrics-bump
• buffer-nr
• bugfix/add-missing-opts-reference-conf
• bugfix/code-visibility
• bugfix/fix-kafka-clients-version
• bugfix/fix-kafka-clients-version-for-ce2
• bugfix/replicator-async-commit-logging
• bugfix/topic-replicator-log-level
• bump-cats-helper
• bump-cats-helper-ce2
• bump-play-json-jsoniter
• cassanda-common
• cassanda-settings-deprecate-setifempty
• cassandra-sync
• ce2-remove-old-scache
• ce2/bincompat-smetrics-bump
• ce2/pointers/query-by-topic+partition
• chore/update-skafka
• chore/update-skafka-for-ce2
• consistency-config
• damper-test-fix-2
• df/DistributeJob-doc
• df/SHRIKE-73-java-kafka-producer-metrics
• df/batch-preserve-offsets
• df/correlate-events-with-meta
• df/down-crease-req-rate-to-pointer2-table
• df/expose-head-cache-config
• df/expose-set-expire-api
• df/make-data-integrity-velidation-optional
• df/pointers2
• df/reimplement-deduplication
• df/segment-and-co-docs
• df/use-kafka-topic-for-pointer
• edubrovskii/compatible-lazy-consumer-pool
• edubrovskii/kafka-consumer-pool
• extract-core
• feature/replicator-CooperativeStickyAssignor
• feature/replicator-async-offset-commit
• feature/scala-3
• feature/testcontainers-kafka
• feature/update-formatting
• fix-damper-test
• fix-insert-order
• fix-insert-order-ce2
• improve-PointerStatements
• improve-PointerStatements-1
• improve-PointerStatements-2
• isurkov/bincompat-smetrics-bump
• isurkov/ce2/bincompat-smetrics-bump
• keyspace-config
• keyspace-config-deprecation
• m/0.0.x-631
• m/4.0.0
• m/636-expose-new-temporary-metric(s)-on-when-pointer-table-is-used-as-fallback
• m/add-explicit-types-everywhere
• m/add-pekko-module
• m/batch-investigation
• m/batch-investigation-reworked-logic
• m/cats-effect-3.6.3
• m/change-package
• m/collect-journal-expiration-values-as-metric
• m/expose-client-version-as-metric
• m/improve-local-build
• m/introduce-usage-of-release-workflow
• m/make-version-non-optional
• m/notes-for-`Next-Developer`
• m/pre-676
• m/prepare-for-5.0.0
• m/prepare-for-pekko-support
• m/remove-deprecated-code
• m/remove-the-fallbacks-and-release-the-version-#637
• m/scalafmt
• m/update-building-infrastructure
• m/update-dependencies
• m/update-scassandra-to-5.2.1
• m/upgrade-warnings-to-errors
• m/use-Oracle-JDK-for-build
• m/use-nicer-package-for-pekko-artifacts
• m/version
• master
• migrate-schema
• more-scaladocs
• more-scaladocs-10
• more-scaladocs-11
• more-scaladocs-12
• more-scaladocs-13
• more-scaladocs-14
• more-scaladocs-15
• more-scaladocs-16
• more-scaladocs-17
• more-scaladocs-18
• more-scaladocs-19
• more-scaladocs-2
• more-scaladocs-20
• more-scaladocs-21
• more-scaladocs-22
• more-scaladocs-23
• more-scaladocs-24
• more-scaladocs-4
• more-scaladocs-5
• more-scaladocs-6
• more-scaladocs-7
• more-scaladocs-8
• more-scaladocs-9
• pointers/always-query-by-topic-and-partition
• pointers/drop-select-all-partitions-api
• re-implement_purge_delete
• refs/tags/v0.3.13
• refs/tags/v0.4.0
• remove-old-scache
• revert-547-update/smetrics-2.2.0
• row-was-applied
• scalafix-config
• series/0.0.x
• settings-config
• simplify-create-schema
• snapshot
• snapshot-class
• snapshot-record
• snapshot-store
• split-schema-creation
• testcontainers-cassandra
• testcontainers-cassandra-ce2
• update-akka-serialization-docs
• update-sbt
• update-scache
• update-scache-ce2
• update-scala
• update-scassandra
• update/akka-test-actor-0.3.0
• update/cassandra-1.21.0
• update/cassandra-1.21.1
• update/cassandra-1.21.2
• update/cassandra-1.21.3
• update/cats-core-2.13.0
• update/cats-helper-3.12.0
• update/cats-helper-3.12.2
• update/circe-core-0.14.12
• update/circe-core-0.14.13
• update/circe-core-0.14.14
• update/executor-tools-1.0.5
• update/hostname-0.3.0
• update/log4j-over-slf4j-2.0.17
• update/logback-classic-1.5.13
• update/logback-classic-1.5.14
• update/logback-classic-1.5.15
• update/logback-classic-1.5.16
• update/logback-classic-1.5.17
• update/logback-classic-1.5.18
• update/pekko-extension-test-actor-0.0.4
• update/pekko-persistence-1.1.5
• update/pekko-persistence-1.2.0
• update/play-json-2.10.7
• update/play-json-3.0.5
• update/play-json-jsoniter-1.2.3
• update/pureconfig-core-0.17.9
• update/sbt-1.10.10
• update/sbt-1.10.11
• update/sbt-1.10.7
• update/sbt-1.10.8
• update/sbt-1.11.0
• update/sbt-1.11.1
• update/sbt-1.11.3
• update/sbt-1.11.4
• update/sbt-1.11.5
• update/sbt-1.11.6
• update/sbt-coveralls-1.3.15
• update/sbt-dynver-5.1.1
• update/sbt-scalafix-0.14.0
• update/sbt-scalafix-0.14.2
• update/sbt-scalafmt-2.5.3
• update/sbt-scalafmt-2.5.4
• update/sbt-scoverage-2.3.0
• update/sbt-scoverage-2.3.1
• update/scache-5.1.3
• update/scala3-library-3.3.6
• update/scalafmt-core-3.8.4
• update/scalafmt-core-3.8.5
• update/scalafmt-core-3.8.6
• update/scalafmt-core-3.9.0
• update/scalafmt-core-3.9.1
• update/scalafmt-core-3.9.10
• update/scalafmt-core-3.9.2
• update/scalafmt-core-3.9.3
• update/scalafmt-core-3.9.5
• update/scalafmt-core-3.9.6
• update/scalafmt-core-3.9.7
• update/scalafmt-core-3.9.8
• update/scalafmt-core-3.9.9
• update/scodec-bits-1.2.1
• update/scodec-bits-1.2.2
• update/scodec-bits-1.2.3
• update/scodec-bits-1.2.4
• update/scodec-core-1.11.11
• update/scodec-core-2.3.3
• update/skafka-17.1.4
• update/skafka-17.2.0
• update/skafka-17.2.1
• update/skafka-17.2.2
• update/smetrics-2.3.0
• update/smetrics-2.3.2
• update/testcontainers-scala-cassandra-0.41.5
• update/testcontainers-scala-cassandra-0.41.8
• update/testcontainers-scala-cassandra-0.43.0
• v3.2.1
• v3.2.2
• v3.2.3
• v3.2.4
• v3.3.0
• v3.3.1
• v3.3.2
• v3.3.3
• v3.3.4
• v3.3.5
• v3.3.6
• v3.3.7
• v3.3.8
• v3.3.9
• v3.4.0
• v3.4.1
• v3.4.2
• v3.5.0
• v3.5.1
• v3.6.0
• v4.0.0
• v4.0.1
• v4.0.2
• v4.1.0
• v4.1.1
• v4.1.2
• v4.1.3
• v4.1.4
• v4.1.5
• v4.1.6
• v4.1.7
• v4.1.8
• v4.2.0
• v4.2.1
• v4.3.0
• v4.3.1
• v5.0.0
• v5.0.0-RC1
• v5.0.0-RC2
• v5.0.0-RC3
• v5.0.0-RC4
• v5.0.0-RC5
• wip/scala-3
• yklymko/kafka-consumer-pool

Build # 13240311130

Build Type

Pull #720

Committed by migesok

Commit Message

Add ReplicatedOffsetNotifier callback

Current recovery logic doesn't scale when the volume of events is large,
because clients have to consume all the events on all the app nodes.
Additionally, current logic creates Kafka consumers on-demand,
which is a heavy operation. This consumer creation happens in spikes,
when replicator is running late, and affects client applications stability.

Introducing a side channel for notifications about replication progress
will allow us in the future to have an alternative implementation for recovery
without these flaws. I.e. notifications go to a separate lean Kafka topic
which is read by all client nodes. On recovery:
- a mark is written
- wait until mark offset confirmed replicated in the notification topic
  (with potential fallback to polling of the pointer table in Cassandra)
- recover from Cassandra only

Pull Request Pull Request #720: Add ReplicatedOffsetNotifier callback

Run Details

6 of 7 new or added lines in 3 files covered. (85.71%)

14 existing lines in 12 files now uncovered.

3164 of 3816 relevant lines covered (82.91%)

1.39 hits per line