apache / bookkeeper / 425

72%

push

jenkins

ISSUE #1967: make ledger creation and removal robust to zk connectionloss


Descriptions of the changes in this PR:

The bookkeeper project ZooKeeperClient wrapper for the ZooKeeper client
will resend zk node creations and removals upon reconnect after a
ConnectionLoss event. In the event that the original succeeded, the
resent operation will erroneously return LedgerExistException or
NoSuchLedgerExistsException for creation and removal respectively.

For removal, this patch limits the operation by allowing it to always
succeed if the ledger does not exist in order to make it idempotent.
This is appears to be the simplest solution as exclusive removal isn't
important.

**Note, the above is an actual change to the bk client semantics**

For creation, exclusive creation is cleary important for correctness,
so this patch adds a creator token field to the LedgerMetdata to
disambiguate the above race from a real race. For
AbstractZkLedgerManager, this is simply a random long value.

There's an oportunity for optimization with the above if exclusive
ledger creation failures are expected to be common.  You only actually
need to perform this check if the operation was really resent.  I chose
not to go this route yet because it would require messing with the
ZooKeeperClient interface to surface that information without burdening
other callers.

If the client is set to version 2 or older, this field will be ignored
and the old behavior will be retained.  If the client is version 3 or
newer but creation races with an older client, the new client will
interpret the nonce to be BLANK and thereby detect the race correctly.



Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Sijie Guo <sijie@apache.org>

This closes #2006 from reddycharan/zkretrialrobust, closes #1967

33 of 33 new or added lines in 4 files covered. (100.0%)

26706 of 37514 relevant lines covered (71.19%)

0.71 hits per line