swiftstack / container-crawler / 176

89%

push

travis-ci

Add support for crawling sharded containers (#26)

Add support for crawling sharded containers.

Previously, sharded containers could not be processed by the container
crawler, as it did not know where the shard databases are located. This
change allows container crawler to properly handle such containers by
finding all of the shards.

The patch checks for sharded containers in two different ways.  If the 
container database is local to the node, the container broker is used to
get the container metadata and thus the shard status. This does not 
require any HTTP requests to Swift services.

There could be a scenario where a shard of a container might not be
stored on the same node as the root container, which would mean that on
that particular node we would never get the broker of the root
container.  For this particular case, we use Swift's `internal_client`
to get the container metadata and check the shard status. If a container
is sharded, we will send an account listing request.

In the future, we will consider writing down the sharded state of a
container to limit the account server queries.

113 of 128 branches covered (88.28%)

Branch coverage included in aggregate %.

296 of 327 relevant lines covered (90.52%)

0.91 hits per line