ipfs / ipfs-cluster / 3096

65%

master: 65%

push

travis-ci

Sharding: Fix unpinning of common shards

As documented in #496, unpinning a shard which is part of multiple
cluster-DAGs was not handled.

Before, we had a back link to the ClusterDAGs (.Parents) in the shards,
but this implied patching (re-pinning) all shard objects created at the
end of the adding process when the ClusterDAGPin is created and known.

Alternatively, we could make shards unique, but this defeats the property
that if a shard is already allocated somewhere, we should not re-store its
parts on a potentially different allocation because we cannot distinguish
they are the same.

Another approach, taken here, is to count all the times shard is part of a
clusterDAG when unpinning. This requires reading all the ClusterDAGs blocks and
writing down their shards counting how many times they are referenced.
**This may not scale well**. The main part of this PR is the end-to-end
tests for sharding, which should still work regardless of the approach.

Issues with this approach: unpinning will be slow when:

* The state has many items (we need to loop all of them and pick ClusterDAG pins)
* The state has many clusterDAG pins (we need to read the blocks from IPFS,
  parse them and loop the Links() (shards).
* The clusterDAGs have many shards. We loop the shard CIDs and put them
  into a map to count how many times they are referenced. Many shards
  will potentially make a big map.
* Unpinning multiple things affecting the same shards at the same time? Might as well leave
  things pinned.

Issues with the having-a-Parents-reference-in-every-shard approach:

* We need to re-pin shards at the end of adding
* Potentially a big mess if things go wrong.
* Unpinning is easier

Other approaches:

* Counting the references in the pin (carrying a number instead of Parents)
  * Would not need shard re-pinning (pinning a shard-pin that exists just increases count + 1)
  * Would not need looping the state for unpin
  * Adds one more field to the pin type (space)... (continued)

58 of 58 new or added lines in 3 files covered. (100.0%)

6341 of 9769 relevant lines covered (64.91%)

104.39 hits per line