ipfs / ipfs-cluster
65%
master: 65%

LAST BUILD BRANCH: feat/expvar
DEFAULT BRANCH: master
Repo Added
Files 37
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH fix/post-sharding
branch: fix/post-sharding
CHANGE BRANCH
x

coverage: 64.909% (-0.5%) from 65.374%
3096

push

travis-ci

hsanjuan 
Sharding: Fix unpinning of common shards

As documented in #496, unpinning a shard which is part of multiple
cluster-DAGs was not handled.

Before, we had a back link to the ClusterDAGs (.Parents) in the shards,
but this implied patching (re-pinning) all shard objects created at the
end of the adding process when the ClusterDAGPin is created and known.

Alternatively, we could make shards unique, but this defeats the property
that if a shard is already allocated somewhere, we should not re-store its
parts on a potentially different allocation because we cannot distinguish
they are the same.

Another approach, taken here, is to count all the times shard is part of a
clusterDAG when unpinning. This requires reading all the ClusterDAGs blocks and
writing down their shards counting how many times they are referenced.
**This may not scale well**. The main part of this PR is the end-to-end
tests for sharding, which should still work regardless of the approach.

Issues with this approach: unpinning will be slow when:

* The state has many items (we need to loop all of them and pick ClusterDAG pins)
* The state has many clusterDAG pins (we need to read the blocks from IPFS,
  parse them and loop the Links() (shards).
* The clusterDAGs have many shards. We loop the shard CIDs and put them
  into a map to count how many times they are referenced. Many shards
  will potentially make a big map.
* Unpinning multiple things affecting the same shards at the same time? Might as well leave
  things pinned.

Issues with the having-a-Parents-reference-in-every-shard approach:

* We need to re-pin shards at the end of adding
* Potentially a big mess if things go wrong.
* Unpinning is easier

Other approaches:

* Counting the references in the pin (carrying a number instead of Parents)
  * Would not need shard re-pinning (pinning a shard-pin that exists just increases count + 1)
  * Would not need looping the state for unpin
  * Adds one more field to the pin type (space)... (continued)

58 of 58 new or added lines in 3 files covered. (100.0%)

6341 of 9769 relevant lines covered (64.91%)

104.39 hits per line

Relevant lines Covered
9769 RELEVANT LINES 6341 COVERED LINES
104.39 HITS PER LINE
Source Files on fix/post-sharding
Coverage ∆ File Lines Relevant Covered Missed Hits/Line

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
3096 fix/post-sharding Sharding: Fix unpinning of common shards As documented in #496, unpinning a shard which is part of multiple cluster-DAGs was not handled. Before, we had a back link to the ClusterDAGs (.Parents) in the shards, but this implied patching (re-pinni... push hsanjuan travis-ci
64.91
3001 fix/post-sharding Sharding: Fix unpinning of common shards As documented in #496, unpinning a shard which is part of multiple cluster-DAGs was not handled. Before, we had a back link to the ClusterDAGs (.Parents) in the shards, but this implied patching (re-pinni... push hsanjuan travis-ci
65.37
See All Builds (2373)