istresearch / scrapy-cluster / 228 / 1
71%
master: 66%

LAST BUILD BRANCH: dev
DEFAULT BRANCH: master
Ran
Files 37
Run time 2s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 65.295%. First build
centos=1 distribution=centos version=7 init=/usr/lib/systemd/systemd run_opts=--privileged --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro

push

travis-ci

web-flow 
Add Docker Support (#72)

* Beginning work on docker branch.

Work here will focus on creating Dockerfiles for the crawler, kafka monitor, and redis monitor. Will use supervisord as a base with environment variables to alter/update any kind of configuration we need within either the python settings.py files or the supervisor configs.

references #48 #26

* Kafka Monitor Docker

Adds a dockerfile to setup and run the kafka monitor. I do note that there are settings and configuration within that tie it into the docker-compose file, but I think they can be fairly easily overridden and maybe I switch them to just using the raw `settings.py` file instead of `localsettings.py`.

This uses stand alone containers for Kafka, Zookeeper, and Redis, and each of the smaller scrapy cluster components
will be their own separate containers as well.

Everything will run under supervisord so you can scale the number of processes both within the container and outside. There is much more work to be done, for example reading in environment variables to override things in the settings file, configuring docker hub, etc.

* Added Redis Monitor to docker setup.

This is an initial cut at making the redis monitor docker compatible. There are still many environmen
t variables to specify but this gets us prett far along

In testing this I also found a dormant issue with ensuring the ZK file path existed, which is easy enough to fix and will be merged in when this branch is decently complete

* Added Dockerfile for crawler

This commit adds the link spider to be compatible with Docker. This completes the initial cut of dockerization of the three core components, and todo is the following:

- Define environment variable overrides for commonly altered configurations
- Documentation under the Advanced Topics as well as Quickstart to supply yet another alternative to working and provisioning Scrapy Cluster.
- Update Changelog
- Merge branch back into dev.... (continued)

1650 of 2527 relevant lines covered (65.29%)

0.65 hits per line

Source Files on job 228.1 (centos=1 distribution=centos version=7 init=/usr/lib/systemd/systemd run_opts=--privileged --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro)