SOHU-Co / kafka-node / 1136 / 15

89%

master: 89%

cron

travis-ci

Multiple fixes to message ordering and compression, Fixes #298 (#1072)

* Multiple fixes to message ordering and compression, Fixes #298

There was numerous risks with the fetch process resolved here.

1) Compressed messages, as mentioned in #298, would be guaranteed out of order
We resolve this by making fetch response buffering be handled through a chain
of executions, where one partition is not emitted until the previous one is done,
and able to be processed asynchronously.

This however introduces new risk as the system only dealt with synchronous
processing before, meaning 2 fetch buffers could not be processing at the same
time. But because it's now async, we could end up processing multiple.

We now need to guard starting a new fetch while one is still processing, to ensure
that a new fetch does not begin with stale offset data.

2) Fetch requests may start, and end up being long polling. This fetch result may
end up coming back after a rebalance, or disconnection from the group occurred.

Those fetch requests would then emit events, which may be at different
offsets, topics, or partitions than the new consumer group membership is responsible for...

So if you received an old fetch response for a topic/partition you are no longer
responsible for, you would receive errors. Or even worse, if you are still responsible
for this partition, you may receive a message that about to be received in the new
memberships fetch request.

We will now run a state validator on the message handler to ensure our group
membership has not changed since the fetch request started, and ignore the response
if it has.

3) Kafka, for some unknown reasons, and one known, in the case of compression,
can return an offset older than the offset you requested.

This is a message we know we have already emitted if our topic payloads already
contained this offset.

We need to skip this message, and any message that shouldn't of been in the re... (continued)

1278 of 1691 branches covered (75.58%)

3942 of 4420 relevant lines covered (89.19%)

7186.92 hits per line