kafkaex / kafka_ex / 583

83%

Pull #276

travis-ci

Send heartbeat earlier

When using a kafka_ex client that was connecting over a high-latency
connection, we would occasionally start getting the following:

```
10:55:27.027 [warn]  Received error :unknown_member_id, consumer group
manager will exit regardless.

10:55:27.028 [error] GenServer #PID<0.280.0> terminating
** (RuntimeError) Error joining consumer group klutzy:
:unknown_member_id
    (kafka_ex) lib/kafka_ex/consumer_group/manager.ex:224:
KafkaEx.ConsumerGroup.Manager.join/1
    (kafka_ex) lib/kafka_ex/consumer_group/manager.ex:163:
KafkaEx.ConsumerGroup.Manager.handle_info/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: :heartbeat
```

By default, the heartbeat_interval is 5 seconds, and we initiate the
timer after the join/sync is complete. This means we have
`session_timeout - heartbeat_interval` time to actually join the group.
If latencies are high (tested with 1s latency), it consistently times
out before finishing the join group - even with no messages to process.

A workaround is to use a heartbeat_interval that is very low, say 500ms,
but this just sends a lot of messages over a high-latency connection,
which may not be desireable.

This change sends a timeout immediately after completing the join/sync cycle so
that we don't have to set the heartbeat_interval to an extremely low
value. In my testing using toxiproxy to introduce latency in the kafka
connection, this seems to fix the join timeout issue, at least for this case.

1 of 1 new or added line in 1 file covered. (100.0%)

562 of 657 relevant lines covered (85.54%)

2417.7 hits per line