tarantool / crud / 33089793479

89%

master: 89%

push

github

fix: retry a call only after a recovery action

call_with_retry_and_recovery() retried every failed call once, no
matter what the error was, and for a single call it followed the
destination reported in a bucket ref error. Two problems came out of
that.

First, the destination redirect was unsafe. During a rebalancing the
source storage rejects a request with a bucket ref error and reports
the transfer destination; the router then sent the retry straight to
that destination, which may not have received the bucket yet. In fast
mode the request was applied there against an empty space, so the same
key could end up on two storages at once.

Second, a call that failed for an unrelated reason -- a storage error,
a timeout -- was retried blindly, which could silently double the
effective request timeout the user had asked for.

Rework the recovery into an explicit step: recover_from_err() performs
an action for the errors it knows and only then reports whether a
retry makes sense.

  * MISSING_MASTER -- the master is not discovered yet, so it is
    discovered explicitly and the same replicaset is retried.
  * NON_MASTER -- the cached master is stale, but the bucket did not
    move, so the master is updated and the same replicaset is retried.
  * WRONG_BUCKET, BUCKET_IS_LOCKED, TRANSFER_IS_IN_PROGRESS -- the
    route is stale, so the bucket route cache is reset and the retry
    target is re-discovered via vshard:route() instead of being taken
    from the error. If the error covers more than one bucket, there is
    no single target to retry on, so the error is returned as is.
  * Anything else is returned to the caller without a retry.

The retry no longer extends the request timeout: a deadline is taken
before the first attempt, and the retry gets only the time left before
it (request_timeout is clamped to the same value).

Map calls do not go through the recovery at all: they are async, so
storage errors arrive later in the future payload and there is... (continued)

37 of 43 new or added lines in 1 file covered. (86.05%)

5440 of 6143 relevant lines covered (88.56%)

11967.1 hits per line