evolution-gaming / kafka-journal

81%

master: 94%

LAST BUILD ON BRANCH v3.6.0

branch: v3.6.0

CHANGE BRANCH

x

Reset

• v3.6.0
• add-minor-documentation-2
• add-snapshot-projects
• binary-compatibility
• binary-compatibility1
• binary-compatibility10
• binary-compatibility11
• binary-compatibility12
• binary-compatibility13
• binary-compatibility2
• binary-compatibility3
• binary-compatibility4
• binary-compatibility5
• binary-compatibility6
• binary-compatibility7
• binary-compatibility8
• binary-compatibility9
• bincompat-smetrics-bump
• buffer-nr
• bugfix/add-missing-opts-reference-conf
• bugfix/code-visibility
• bugfix/fix-kafka-clients-version
• bugfix/fix-kafka-clients-version-for-ce2
• bugfix/replicator-async-commit-logging
• bugfix/topic-replicator-log-level
• bump-cats-helper
• bump-cats-helper-ce2
• bump-play-json-jsoniter
• cassanda-common
• cassanda-settings-deprecate-setifempty
• cassandra-sync
• ce2-remove-old-scache
• ce2/bincompat-smetrics-bump
• ce2/pointers/query-by-topic+partition
• chore/update-skafka
• chore/update-skafka-for-ce2
• consistency-config
• damper-test-fix-2
• df/DistributeJob-doc
• df/SHRIKE-73-java-kafka-producer-metrics
• df/batch-preserve-offsets
• df/correlate-events-with-meta
• df/down-crease-req-rate-to-pointer2-table
• df/expose-head-cache-config
• df/expose-set-expire-api
• df/make-data-integrity-velidation-optional
• df/pointers2
• df/reimplement-deduplication
• df/segment-and-co-docs
• df/use-kafka-topic-for-pointer
• edubrovskii/compatible-lazy-consumer-pool
• edubrovskii/kafka-consumer-pool
• extract-core
• feature/replicator-CooperativeStickyAssignor
• feature/replicator-async-offset-commit
• feature/scala-3
• feature/testcontainers-kafka
• feature/update-formatting
• fix-damper-test
• fix-insert-order
• fix-insert-order-ce2
• improve-PointerStatements
• improve-PointerStatements-1
• improve-PointerStatements-2
• isurkov/bincompat-smetrics-bump
• isurkov/ce2/bincompat-smetrics-bump
• keyspace-config
• keyspace-config-deprecation
• m/0.0.x-631
• m/4.0.0
• m/636-expose-new-temporary-metric(s)-on-when-pointer-table-is-used-as-fallback
• m/add-explicit-types-everywhere
• m/add-pekko-module
• m/batch-investigation
• m/batch-investigation-reworked-logic
• m/cats-effect-3.6.3
• m/change-package
• m/collect-journal-expiration-values-as-metric
• m/expose-client-version-as-metric
• m/improve-local-build
• m/introduce-usage-of-release-workflow
• m/make-version-non-optional
• m/notes-for-`Next-Developer`
• m/pre-676
• m/prepare-for-5.0.0
• m/prepare-for-pekko-support
• m/remove-deprecated-code
• m/remove-the-fallbacks-and-release-the-version-#637
• m/scalafmt
• m/update-building-infrastructure
• m/update-dependencies
• m/update-scassandra-to-5.2.1
• m/upgrade-warnings-to-errors
• m/use-Oracle-JDK-for-build
• m/use-nicer-package-for-pekko-artifacts
• m/version
• master
• migesok/replicated-offset-notifier
• migrate-schema
• more-scaladocs
• more-scaladocs-10
• more-scaladocs-11
• more-scaladocs-12
• more-scaladocs-13
• more-scaladocs-14
• more-scaladocs-15
• more-scaladocs-16
• more-scaladocs-17
• more-scaladocs-18
• more-scaladocs-19
• more-scaladocs-2
• more-scaladocs-20
• more-scaladocs-21
• more-scaladocs-22
• more-scaladocs-23
• more-scaladocs-24
• more-scaladocs-4
• more-scaladocs-5
• more-scaladocs-6
• more-scaladocs-7
• more-scaladocs-8
• more-scaladocs-9
• pointers/always-query-by-topic-and-partition
• pointers/drop-select-all-partitions-api
• re-implement_purge_delete
• refs/tags/v0.3.13
• refs/tags/v0.4.0
• remove-old-scache
• revert-547-update/smetrics-2.2.0
• row-was-applied
• scalafix-config
• series/0.0.x
• settings-config
• simplify-create-schema
• snapshot
• snapshot-class
• snapshot-record
• snapshot-store
• split-schema-creation
• testcontainers-cassandra
• testcontainers-cassandra-ce2
• update-akka-serialization-docs
• update-sbt
• update-scache
• update-scache-ce2
• update-scala
• update-scassandra
• update/akka-test-actor-0.3.0
• update/cassandra-1.21.0
• update/cassandra-1.21.1
• update/cassandra-1.21.2
• update/cassandra-1.21.3
• update/cats-core-2.13.0
• update/cats-helper-3.12.0
• update/cats-helper-3.12.2
• update/circe-core-0.14.12
• update/circe-core-0.14.13
• update/circe-core-0.14.14
• update/executor-tools-1.0.5
• update/hostname-0.3.0
• update/log4j-over-slf4j-2.0.17
• update/logback-classic-1.5.13
• update/logback-classic-1.5.14
• update/logback-classic-1.5.15
• update/logback-classic-1.5.16
• update/logback-classic-1.5.17
• update/logback-classic-1.5.18
• update/pekko-extension-test-actor-0.0.4
• update/pekko-persistence-1.1.5
• update/pekko-persistence-1.2.0
• update/play-json-2.10.7
• update/play-json-3.0.5
• update/play-json-jsoniter-1.2.3
• update/pureconfig-core-0.17.9
• update/sbt-1.10.10
• update/sbt-1.10.11
• update/sbt-1.10.7
• update/sbt-1.10.8
• update/sbt-1.11.0
• update/sbt-1.11.1
• update/sbt-1.11.3
• update/sbt-1.11.4
• update/sbt-1.11.5
• update/sbt-1.11.6
• update/sbt-coveralls-1.3.15
• update/sbt-dynver-5.1.1
• update/sbt-scalafix-0.14.0
• update/sbt-scalafix-0.14.2
• update/sbt-scalafmt-2.5.3
• update/sbt-scalafmt-2.5.4
• update/sbt-scoverage-2.3.0
• update/sbt-scoverage-2.3.1
• update/scache-5.1.3
• update/scala3-library-3.3.6
• update/scalafmt-core-3.8.4
• update/scalafmt-core-3.8.5
• update/scalafmt-core-3.8.6
• update/scalafmt-core-3.9.0
• update/scalafmt-core-3.9.1
• update/scalafmt-core-3.9.2
• update/scalafmt-core-3.9.3
• update/scalafmt-core-3.9.5
• update/scalafmt-core-3.9.6
• update/scalafmt-core-3.9.7
• update/scalafmt-core-3.9.8
• update/scalafmt-core-3.9.9
• update/scodec-bits-1.2.1
• update/scodec-bits-1.2.2
• update/scodec-bits-1.2.3
• update/scodec-bits-1.2.4
• update/scodec-core-1.11.11
• update/scodec-core-2.3.3
• update/skafka-17.1.4
• update/skafka-17.2.0
• update/skafka-17.2.1
• update/skafka-17.2.2
• update/smetrics-2.3.0
• update/smetrics-2.3.2
• update/testcontainers-scala-cassandra-0.41.5
• update/testcontainers-scala-cassandra-0.41.8
• update/testcontainers-scala-cassandra-0.43.0
• v3.2.1
• v3.2.2
• v3.2.3
• v3.2.4
• v3.3.0
• v3.3.1
• v3.3.2
• v3.3.3
• v3.3.4
• v3.3.5
• v3.3.6
• v3.3.7
• v3.3.8
• v3.3.9
• v3.4.0
• v3.4.1
• v3.4.2
• v3.5.0
• v3.5.1
• v4.0.0
• v4.0.1
• v4.0.2
• v4.1.0
• v4.1.1
• v4.1.2
• v4.1.3
• v4.1.4
• v4.1.5
• v4.1.6
• v4.1.7
• v4.1.8
• v4.2.0
• v4.2.1
• v4.3.0
• v4.3.1
• v5.0.0
• v5.0.0-RC1
• v5.0.0-RC2
• v5.0.0-RC3
• v5.0.0-RC4
• v5.0.0-RC5
• wip/scala-3
• yklymko/kafka-consumer-pool

Build # 11017779211

Build Type

push

Committed by web-flow

Commit Message

Add meta_record_id into journal & record_id metajournal tables for gaining correlation between them (#665)

### Problem
KJ can be in inconsistent state when (after purging) meta from
`metajournal` was deleted while events from `journal` left behind. By
itself its not an issue, but after the journal used again and new meta
inserted - then old events will be available (probably with duplicated
`seqNr`).

### Solution 
Create correlation between meta and events that belong to it, i. e. were
created while the meta was present. Then, on reading journal, only
events that correlate with meta must be used.

### Implementation 
Add `meta_record_id` column to `journal` and `record_id` column to
`metajournal` tables. In appending events `RecordId` from meta will be
used (if meta has it) and if meta is not present yet - new `RecordId`
generated. On reading compare `RecordId` from meta and event ignoring
events with another value. `RecordId` is optional value thus validation
takes place only if its present in meta.

Co-authored-by: Denys Fakhritdinov <dfakhritdinov@evolution.com>

Run Details

49 of 56 new or added lines in 7 files covered. (87.5%)

14 existing lines in 12 files now uncovered.

3280 of 4059 relevant lines covered (80.81%)

1.33 hits per line