cloudmarker / cloudmarker

90%

master: 84%

LAST BUILD ON BRANCH pluginframework

branch: pluginframework

CHANGE BRANCH

x

Reset

• pluginframework
• 0.1.0
• alertdocs
• apidocindent
• audit-version
• auditemails
• azcloud
• azstorage
• azurecloud
• azurecloud-exception
• azvm_record_type
• azwebapp
• badges
• baseconfig
• bracket
• buckets
• build-matrix
• changelog
• clioptions
• clouderror
• common-plugins-config
• consistency
• copy-ext
• copyext
• cov_clean
• dependabot/pip/cryptography-3.3.2
• dependabot/pip/httplib2-0.18.0
• dependabot/pip/httplib2-0.19.0
• dependabot/pip/pyyaml-5.4
• dependabot/pip/rsa-4.7
• dependabot/pip/urllib3-1.26.5
• dist
• docfixes
• doctest-fix
• email_debug_param_from_config
• emailbody
• emaildocs
• empty-port
• esstore
• firewalleventtest
• firewallnorm
• firewallruleevent
• fix-esstore-done
• fix-unused-input-queue
• fix_cloud_worker
• fixbuild
• fixcoverage
• fixindent
• freeze
• gcpcloudlogs
• gcpcloudtest
• iam_policy
• ignore-virtual-machines
• ioworkers
• key_file_name_fix
• lint
• load_yaml_safely
• log
• logfixes
• logger-defaults
• logging-levels
• master
• maxrecs
• mock-records
• mockplugins
• mongodbdefaults
• mongodbfix
• mongodbtests
• multiprocessing
• pluginloader
• readable-version
• readme
• remove-email-config
• removefirewallevent
• rename_logger
• renameparams
• requirements
• rest
• reuse-mock
• rm-venv
• slackalert
• smtpssl
• sphinx
• sphinx-rst-update
• splunkfixes
• splunkhecstore_docs
• splunkhectokentest
• splunkstore
• sub_id_rename
• tests
• travis
• userconfig
• userdoc
• userecordterm
• verdev
• webapp
• worker-com-fields
• worker-error-handling
• workerlogs

Build # 900

Build Type

Pull #147

travis-ci-com

Committed by web-flow

Commit Message

Instantiate plugins in worker processes

Prior to this change, the plugin classes are instantiated in the main
process running `manager.py` and the instantiated plugins are passed to
the respective worker processes implemented in `workers.py` via
`multiprocessing.Queue` objects.

Instantiating a plugin in the main process in `manager.py` and then
passing it to a worker process via a queue is problematic. If a plugin
is not fork-safe, it may not function correctly when used inside a
worker process. One such problem was found with the `MongoDBStore`
plugin which is not fork-safe because it in turn instantiates
`MongoClient` which is not fork-safe. The following warning is
encountered while using `MongoDBStore`:

    UserWarning: MongoClient opened before fork. Create MongoClient only
    after forking. See PyMongo's documentation for details:
    http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
    "MongoClient opened before fork. Create MongoClient only "

There is another possible problem too. If a plugin is not picklable
(serialiazble), then attempting to pass an instantiated plugin over a
`multiprocessing.Queue` object would lead to `PickleError`.

To prevent these issues, this change implements a new strategy for
instantiating the plugins. The main process running `manager.py` passes
the plugin config dictionaries to the worker processes over
`multiprocessing.Queue` objects. The worker processes instantiate the
plugins from the plugin config dictionaries. This ensures that we pass
only a serializable object (i.e., a dictionary, not an instantiated
plugin) over the queue. It also ensures that a plugin is instantiated in
the same process where it is used, i.e., within a worker process.

Pull Request Pull Request #147: Instantiate plugins in worker processes

Run Details

439 of 494 branches covered (88.87%)

Branch coverage included in aggregate %.

39 of 39 new or added lines in 3 files covered. (100.0%)

2075 of 2298 relevant lines covered (90.3%)

2.71 hits per line