cloudmarker / cloudmarker / 901

84%

push

travis-ci-com

Instantiate plugins in worker processes

Prior to this change, the plugin classes are instantiated in the main
process running `manager.py` and the instantiated plugins are passed to
the respective worker processes implemented in `workers.py` via
`multiprocessing.Queue` objects.

Instantiating a plugin in the main process in `manager.py` and then
passing it to a worker process via a queue is problematic. If a plugin
is not fork-safe, it may not function correctly when used inside a
worker process. One such problem was found with the `MongoDBStore`
plugin which is not fork-safe because it in turn instantiates
`MongoClient` which is not fork-safe. The following warning is
encountered while using `MongoDBStore`:

    UserWarning: MongoClient opened before fork. Create MongoClient only
    after forking. See PyMongo's documentation for details:
    http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
    "MongoClient opened before fork. Create MongoClient only "

There is another possible problem too. If a plugin is not picklable
(serialiazble), then attempting to pass an instantiated plugin over a
`multiprocessing.Queue` object would lead to `PickleError`.

To prevent these issues, this change implements a new strategy for
instantiating the plugins. The main process running `manager.py` passes
the plugin config dictionaries to the worker processes over
`multiprocessing.Queue` objects. The worker processes instantiate the
plugins from the plugin config dictionaries. This ensures that we pass
only a serializable object (i.e., a dictionary, not an instantiated
plugin) over the queue. It also ensures that a plugin is instantiated in
the same process where it is used, i.e., within a worker process.

439 of 494 branches covered (88.87%)

Branch coverage included in aggregate %.

2075 of 2298 relevant lines covered (90.3%)

2.71 hits per line