tarantool / tarantool / 10079927561

88%

Build Type

push

github

Committed by sergepetrenko

Commit Message

relay: batch sent rows

Relay sends rows one by one, issuing one `write()` system call per each
sent row. This hurts performance quite a lot: in the 1mops_write test
[1] on my machine master node is able to reach ~ 1.9 million RPS while
replication performance peaks at ~ 0.5 million RPS.

As shown by a PoC patch, batching the rows allows to remove the
replication bottleneck almost completely. Even sending 2 rows per one
write() call already doubles the performance, and sending 64 rows per
one write() call allows the replica to catch up with master's speed [2].

This patch makes relay buffer rows until the total buffer size reaches a
certain threshold, and then flush everything with a single writev()
call.

The flush threshold is configurable via a new tweak -
xrow_buf_flush_size, which has a default value of 16 kb.

A size threshold is chosen over simple row count, because it's more
flexible for different row sizes: if the user has multi-kilobyte tuples,
batching hundreds of such tuples might overflow the socket's send buffer
size, resulting in extra wait until the send buffer becomes free again.
This extra time might've been used for enqueuing extra rows. The
problem would become even worse for hundred-kilobyte tuples and so on.

At the same time the user might have different workload types. One with
large tuples and other with relatively small ones, in which case
batching by size rather than by row count is more predictable
performance wise.

[1] https://github.com/tarantool/tarantool/blob/master/perf/lua/1mops_write.lua
[2] https://www.notion.so/tarantool/relay-248de4a9600f4c4d8d83e81cf1104926

Closes #10161

NO_DOC=not a documentable feature
NO_TEST=covered by existing tests

Run Details

67900 of 121042 branches covered (56.1%)

94 of 96 new or added lines in 3 files covered. (97.92%)

251 existing lines in 15 files now uncovered.

100689 of 115551 relevant lines covered (87.14%)

2429268.78 hits per line