istresearch / scrapy-cluster / 228

71%

master: 66%

push

travis-ci

Add Docker Support (#72)

* Beginning work on docker branch.

Work here will focus on creating Dockerfiles for the crawler, kafka monitor, and redis monitor. Will use supervisord as a base with environment variables to alter/update any kind of configuration we need within either the python settings.py files or the supervisor configs.

references #48 #26

* Kafka Monitor Docker

Adds a dockerfile to setup and run the kafka monitor. I do note that there are settings and configuration within that tie it into the docker-compose file, but I think they can be fairly easily overridden and maybe I switch them to just using the raw `settings.py` file instead of `localsettings.py`.

This uses stand alone containers for Kafka, Zookeeper, and Redis, and each of the smaller scrapy cluster components
will be their own separate containers as well.

Everything will run under supervisord so you can scale the number of processes both within the container and outside. There is much more work to be done, for example reading in environment variables to override things in the settings file, configuring docker hub, etc.

* Added Redis Monitor to docker setup.

This is an initial cut at making the redis monitor docker compatible. There are still many environmen
t variables to specify but this gets us prett far along

In testing this I also found a dormant issue with ensuring the ZK file path existed, which is easy enough to fix and will be merged in when this branch is decently complete

* Added Dockerfile for crawler

This commit adds the link spider to be compatible with Docker. This completes the initial cut of dockerization of the three core components, and todo is the following:

- Define environment variable overrides for commonly altered configurations
- Documentation under the Advanced Topics as well as Quickstart to supply yet another alternative to working and provisioning Scrapy Cluster.
- Update Changelog
- Merge branch back into dev.... (continued)

1 of 1 new or added line in 1 file covered. (100.0%)

1650 of 2527 relevant lines covered (65.29%)

1.31 hits per line