apache / bookkeeper / 75 / 1
72%
master: 72%

DEFAULT BRANCH: master
Ran
Files 414
Run time 16s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 72.463% (-0.05%) from 72.515%
75.1

push

jenkins

Sijie Guo 
[TABLE SERVICE] a zk based storage container controller

Descriptions of the changes in this PR:

*Motivation*

The original storage container "controller" was written in helix. The helix controller's placement is kind of not assuming any properties of shared storage. however the table service is leverage bookkeeper's as its segment/log store. the table service is built in a more `stateless` way, where a storage container can be moved between servers in a much lightweight way.

Also helix codebase is a bit large, if we eventually going to eliminate zookeeper, the cost of switching it off helix will be much expensive. so this PR is introducing a simple zk based controller for assigning storage containers to servers.

*Solution*

In this zk base solution, it is comprised of 3 parts: server registration, cluster & storage controller, storage container manager.

**registration**

this solution leverages existing registration client/manager interfaces. so each storage server registers itself under `/stream/servers/available`.

**cluster & storage controller**

A storage server is elected as a the leader, which runs a ClusterController. The cluster controller reads cluster metadata from zookeeper (the cluster metadata includes like how many containers available in the cluster), and use a storage controller to compute an assignment plan as the ideal state for container assignment mapping, and then update the assignment mapping to zookeeper under `/stream/assignment`.

**storage container manager**

Each storage server runs a storage container manager, which watches the assignment plan in zookeeper. The cluster controller updates the assignment plan. Once it receives watches/updates from the assignment plan that computed by cluster controller, the storage container manager will align itself to the ideal assignment map by starting/stopping storage containers.

Author: Sijie Guo <sijie@apache.org>

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Jia Zhai <Non... (continued)

23668 of 32662 relevant lines covered (72.46%)

0.72 hits per line

Source Files on job 75.1