Logflare / logflare / 57fda7065cc6339bd999abab16b4c80b583005c3

81%

Build Type

push

github

Committed by web-flow

Commit Message

perf: pass event ids through broadway pipeline instead of full structs (#3569)

* perf: pass event ids through broadway pipeline instead of full structs

* Remove unused

* Remove unused

* Move process_data to batcher

* Fix test

* Fix test/format

* Handle duplicate ids

* Prevent duplicate scheme update

* Retry and delete stale processing messages, chunk big query writes, optimize size calculation

* Extra oversight for monitor script

* Small refactor, more tests

* Small refactor

* Additional tests

* Include storage api in chunk_by logic

* Additional tests

* Set max_batch_length in context

* Address pr comments. Handling around race conditions.

* Remove tags from telemetry

* Size in bufferprodcuer

* Format

* Fix lint

* Calculate size when adding to the table

---------

Co-authored-by: Adam Mokan <amokan@gmail.com>

Coverage Stats

113 of 123 new or added lines in 4 files covered. (91.87%)

6 existing lines in 3 files now uncovered.

13152 of 16216 relevant lines covered (81.11%)

3871.83 hits per line