zentralopensource / zentral / 30397698567
90%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 1016
Run time 2min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 89.488% (+0.05%) from 89.435%
30397698567

push

github

np5 
Split the SQS batches by size, not only by count

SQS caps a single message and the sum of a send_message_batch at the same
1MiB, and rejects the batch as a whole when it is over. SQSSendThread
batched ten messages with no size accounting, so one big message took the
nine it happened to travel with down with it — and send_entries clears
its entries whatever the outcome, so on the producer path, where there is
no receipt handle to re-enqueue, all ten raw events were simply lost.

Compressing the raw events cut the exposure by about nine but did not
remove it: a machine snapshot tree past ~4,270 apps still comes out over
a tenth of the cap once compressed and base64'd, and ten of those in one
batch are over again. Account for the bytes as entries accumulate and
send what fits before adding one that would cross the cap, so the batch
adapts to the payloads instead of always counting to ten.

A message that is over the cap on its own cannot be sent at all, here or
anywhere else. It now travels alone, so it only takes itself down when
the queue rejects it, and it is logged with its size and routing key.

The message attributes count against the size too, not only the body, so
the routing key is part of the accounting. The budget keeps a margin
under 1MiB: the accounting on the SQS side is not byte for byte the one
in _entry_byte_size, and being a few bytes optimistic would cost a whole
batch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

49768 of 55614 relevant lines covered (89.49%)

0.89 hits per line

Coverage Regressions

Lines Coverage ∆ File
48
30.72
 0.22% zentral/core/queues/backends/google_pubsub/__init__.py
21
65.03
 0.22% zentral/core/queues/backends/kombu.py
19
81.73
 0.06% zentral/core/queues/backends/aws_sns_sqs/__init__.py
Jobs
ID Job ID Ran Files Coverage
1 30397698567.1 1016
89.49
 GitHub Action Run
Source Files on build 30397698567