uber / cadence / 018dc39c-5b2f-40c1-abd2-356e5a779477 / 1
72%
master: 72%

DEFAULT BRANCH: master
Ran
Files 670
Run time 12s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 62.705% (-0.04%) from 62.744%
018dc39c-5b2f-40c1-abd2-356e5a779477.1

push

buildkite

web-flow 
Emits a counter value for every unique view of the hashring (#5672)

Context and problem
The consistent hashring is an eventually consistent way to see the services' members for routing in amongst shards. It's an eventually-consistent system (since actual strong consistency for membership is controlled by the database shard locking).

However, it's possible that it can be very eventually consistent, particularly if the membership resolving system is serving stale data for some reason. This can cause problems because particularly in the case of history, it breaks inter-service communication and routing.

Solution
A dumb way to determine if there's an inconsistency therefore, is to just hash the complete membership set and emit it for each host, like a fingerprint. In the healthy case, this random identifier value will quickly converge across hosts. . In the event their views are inconsistent, this will appear as a different guage values which remain persistently different, indicating that operationally some manual operation must be taken.

querying the data
Assuming m3, the trick will be to look for differences between the identifier values. The value itself is just a hash value and arbitrary. Therefore, selecting differences between the upper bound thusly:

max = fetch service:cadence* operation:hashring region:region1 service:cadence-history | maxSeries;
min = fetch service:cadence* operation:hashring region:region1 service:cadence-history | minSeries;

-- select where they're different
max | != (min)

-- and emit -1 for the default case where the upper value and lower value are all the same
| transformNull -1

92594 of 147666 relevant lines covered (62.71%)

2297.36 hits per line

Source Files on job 018dc39c-5b2f-40c1-abd2-356e5a779477.1