uber / cadence / 01903bc8-798b-41ce-bd5b-c8557de7de2b

72%

Build Type

push

buildkite

Committed by web-flow

Commit Message

Persist failover history in DomainInfo data (#6139)

**What changed?**
Added functionality to persist recent failover event data in the DomainInfo whenever a valid failover is executed.
`FailoverEvent` contains the failover `timestamp`, `fromCluster`, `toCluster`, and `FailoverType`("Force"/"Grace") information.
`FailoverHistory` is the key in the `DomainInfo` `data`. It's a slice stored as a string containing the `FailoverEvents`, with max size defined by `dynamicconfig.FrontendFailoverHistoryMaxSize`. It has a default value of 5 and domain filter allowed.
`FailoverHistory` always keep the n most recent `FailoverEvents` and it's sorted by descending timestamp.

**Why?**
Persist failover information, improving failover visibility to clients and the cadence team.

**How did you test it?**
Unit tests and integration tests. Tested locally, triggering failovers in a multiple Cadence clusters with replication environment.

**Potential risks**
The change does not affect the logic of UpdateDomain. It adds the failover info to the DomainInfo data. The main risk is that we introduce something that can cause the code to panic. Errors while adding the FailoverHistory are logged as warnings and do not return/interrupt the UpdateDomain action.

**Release notes**

**Documentation Changes**

Run Details

27 of 39 new or added lines in 3 files covered. (69.23%)

39 existing lines in 11 files now uncovered.

106671 of 149198 relevant lines covered (71.5%)

2612.88 hits per line