nats-io / nats-server / 35308189495
83%

DEFAULT BRANCH: main
Ran
Jobs 1
Files 66
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 82.763% (+0.2%) from 82.534%
35308189495

push

github

web-flow 
(2.15) [IMPROVED] MQTT: Pipeline QoS2 PUBLISH and PUBREL processing (#8416)

QoS2 ingest was stop-and-wait on the server side. Each PUBLISH blocked
the publisher's readLoop on the JetStream store of its held copy into
`$MQTT_qos2in`, and each PUBREL on three more round trips: load the
copy back, delete it waiting for the ack, and store it into `$MQTT_msgs`
for delivery. A QoS2 publisher was therefore capped at a few hundred
messages per second whatever its in-flight window.

This PR extends the QoS1 ack pipeline of #8415 to QoS2. The JetStream
operations are submitted asynchronously, the PUBREL delivers from a copy
the connection kept so there is no load, the delete no longer waits, and
the PUBREC and PUBCOMP are sent from the pipeline as the acks arrive, in
packet order. Single-connection QoS2 throughput, flat at 600-800/s on
main whatever the window, reaches ~18K/s at a 256-message window (22x)
and rises with the client's window from 1.4x at a window of 1; aggregate
QoS2 over 100-200 connections improves 1.4-1.5x; QoS1 and QoS0 are
within the run's control spread. Where asynchrony costs the readLoop
its read-your-own-writes view of JetStream, the connection keeps the
state it needs in memory, and a `Nats-Msg-Id` on the delivery store
covers the one case a connection cannot see, a PUBREL retransmitted on
another connection.

## Benchmark

GCP, 6 brokers in two 3-node R3 clusters, generators on separate VMs,
two crossover passes (the builds swap clusters between passes), ratios
pooled. `mqtt-test` sweep, QoS2, main (3c80d8f89) vs this branch
(2eb9772b9):

| in flight | main | this PR | ratio |
|---|---|---|---|
| 1 | 579/s (p99 2.2ms) | 801/s (p99 1.6ms) | 1.4x |
| 10 | 807/s (p99 17.3ms) | 5,827/s (p99 3.0ms) | 7.2x |
| 20 | 814/s (p99 30.6ms) | 8,800/s (p99 3.9ms) | 10.8x |
| 256 | 816/s (p99 346.9ms) | 18,051/s (p99 24.1ms) | 22.1x |

Aggregate, 100 and 200 connections publishing QoS2 at once: 1.5x and
1.4x (about 12K/s to 17.3K/s). QoS1 aggregate ... (continued)

71840 of 86802 relevant lines covered (82.76%)

317203.12 hits per line

Coverage Regressions

Lines Coverage ∆ File
397
88.85
 0.3% src/github.com/nats-io/nats-server/server/mqtt.go
273
82.95
 0.16% src/github.com/nats-io/nats-server/server/jetstream_cluster.go
233
86.22
 -0.29% src/github.com/nats-io/nats-server/server/accounts.go
66
67.54
 0.08% src/github.com/nats-io/nats-server/server/reload.go
14
77.5
 -0.26% src/github.com/nats-io/nats-server/server/opts.go
13
81.33
 0.28% src/github.com/nats-io/nats-server/server/memstore.go
11
87.43
 -0.18% src/github.com/nats-io/nats-server/server/consumer.go
10
85.07
 -0.46% src/github.com/nats-io/nats-server/server/events.go
8
78.04
 -0.04% src/github.com/nats-io/nats-server/server/filestore.go
7
73.52
 -0.08% src/github.com/nats-io/nats-server/server/jetstream_api.go
6
93.47
 -0.29% src/github.com/nats-io/nats-server/server/gateway.go
6
89.48
 -0.21% src/github.com/nats-io/nats-server/server/leafnode.go
4
86.29
 0.14% src/github.com/nats-io/nats-server/server/stream.go
2
89.47
 -0.07% src/github.com/nats-io/nats-server/server/server.go
1
92.93
 0.02% src/github.com/nats-io/nats-server/server/client.go
1
84.56
 5.5% src/github.com/nats-io/nats-server/server/raft.go
Jobs
ID Job ID Ran Files Coverage
1 35308189495.1 66
82.76
 GitHub Action Run
Source Files on build 35308189495