uber / cadence / 01905fb8-bb8f-4e40-b7b8-0b7dc522c2e4

72%

Build Type

push

buildkite

Committed by web-flow

Commit Message

Bugfix: replication messaged dropped during host shutdown (#6143)

What changed?

Internal details: CDNC-9597

A user reported some problems during a failover in which a workflow, during a continue-as-new event got dropped during replication silently, without any corresponding DLQ message. We were able to track down the (expected) cause to likely have been a shard movement during that time which triggers several unpleasant edge-conditions with interactions with the following:

A silent dropping of errors due to the shard closing
The in-memory offsets being advanced incorrectly during shard-closed events due to some missing control-flow handling
The GetReplication API call actually being a write call and tracking offsets (perhaps somewhat surprisingly, I really don't like this API).
The GetReplicationCall being called, with these invalid parameters during a shard-closing event.
How did you test it?

Tested locally and with unit tests. Was able to repro the sequence of events mostly with unit-tests.

Run Details

14 of 15 new or added lines in 1 file covered. (93.33%)

34 existing lines in 11 files now uncovered.

104697 of 146557 relevant lines covered (71.44%)

2650.06 hits per line