tarantool / crud / 33235083577

89%

Build Type

push

github

Committed by Satbek

Commit Message

fix: retry a call only after a recovery action

call_with_retry_and_recovery() retried every failed call once, whatever
the error was, and for a single call it followed the destination
reported in a bucket ref error. The destination may not have received
the bucket yet, so in fast mode the request was applied there against
an empty space and the same key could end up on two storages. An
unrelated failure -- a storage error, a timeout -- was retried just as
blindly, silently doubling the requested timeout.

Recovery becomes an explicit step: recover_from_err() performs an
action for the errors it knows and only then reports whether a retry
makes sense. The master is located on MISSING_MASTER and the cached one
is updated on NON_MASTER; on WRONG_BUCKET, BUCKET_IS_LOCKED and
TRANSFER_IS_IN_PROGRESS the route cache is reset and the target is
re-discovered via vshard:route() instead of being taken from the error.
If the action did not help, or the error is anything else, it is
returned to the caller as is. A deadline is taken before the first
attempt, so the retry never runs past the requested timeout.

Map calls recover from MISSING_MASTER only: vshard does not search for
a master on an async call and reports it right away, while storage
errors arrive later, in the future payload -- see #513.

On the test side, the safe mode setup is extracted from
helpers.start_cluster() into helpers.set_safe_mode(), so that a test
can toggle the mode on a running cluster.

Closes #509.

Coverage Stats

49 of 64 new or added lines in 1 file covered. (76.56%)

24 existing lines in 4 files now uncovered.

5459 of 6167 relevant lines covered (88.52%)

12284.56 hits per line