cloudmarker / cloudmarker

90%

master: 84%

LAST BUILD ON BRANCH ioworkers

branch: ioworkers

CHANGE BRANCH

x

Reset

• ioworkers
• 0.1.0
• alertdocs
• apidocindent
• audit-version
• auditemails
• azcloud
• azstorage
• azurecloud
• azurecloud-exception
• azvm_record_type
• azwebapp
• badges
• baseconfig
• bracket
• buckets
• build-matrix
• changelog
• clioptions
• clouderror
• common-plugins-config
• consistency
• copy-ext
• copyext
• cov_clean
• dependabot/pip/cryptography-3.3.2
• dependabot/pip/httplib2-0.18.0
• dependabot/pip/httplib2-0.19.0
• dependabot/pip/pyyaml-5.4
• dependabot/pip/rsa-4.7
• dependabot/pip/urllib3-1.26.5
• dist
• docfixes
• doctest-fix
• email_debug_param_from_config
• emailbody
• emaildocs
• empty-port
• esstore
• firewalleventtest
• firewallnorm
• firewallruleevent
• fix-esstore-done
• fix-unused-input-queue
• fix_cloud_worker
• fixbuild
• fixcoverage
• fixindent
• freeze
• gcpcloudlogs
• gcpcloudtest
• iam_policy
• ignore-virtual-machines
• key_file_name_fix
• lint
• load_yaml_safely
• log
• logfixes
• logger-defaults
• logging-levels
• master
• maxrecs
• mock-records
• mockplugins
• mongodbdefaults
• mongodbfix
• mongodbtests
• multiprocessing
• pluginframework
• pluginloader
• readable-version
• readme
• remove-email-config
• removefirewallevent
• rename_logger
• renameparams
• requirements
• rest
• reuse-mock
• rm-venv
• slackalert
• smtpssl
• sphinx
• sphinx-rst-update
• splunkfixes
• splunkhecstore_docs
• splunkhectokentest
• splunkstore
• sub_id_rename
• tests
• travis
• userconfig
• userdoc
• userecordterm
• verdev
• webapp
• worker-com-fields
• worker-error-handling
• workerlogs

Build # 854

Build Type

Pull #138

travis-ci-com

Committed by web-flow

Commit Message

Use processes and threads in Azure cloud plugins

Both `AzCloud` and `AzVM` now rely on multiprocessing and multithreading
to pull resources concurrently from Azure cloud in order to complete its
scan faster.

All of the multiprocessing and multithreading code involved is put in a
new `ioworkers` module. `AzCloud` and `AzVM` invoke `ioworkers.run()` to
launch multiple worker processes each of which launch multiple worker
threads. The callers, i.e., `AzCloud` and `AzVM`, need to provide this
`ioworkers.run()` function two callback functions:

  - An input function which when invoked yields tuples where each tuple
    defines a unit of work.
  - An output function which works on each tuple yielded by the input
    function, does something with it, and then yields outputs.

The output functions are run concurrently by `ioworkers` in multiple
worker threads within multiple worker processes. This two-level
approach, i.e., worker threads within worker processes, has been chosen
to maximize utilization of CPUs. This approaches provides a good balance
between the following two extremes:

  - Using a large number of processes only would consume too many IPC
    resources and increase the overhead of context switching for
    processes.
  - Using a large number of threads only does not scale well in Python
    due to global interpreter lock (GIL) which allows only one thread to
    execute at a time.

Lauching multiple workers sidesteps GIL and lets us utilize multiple
CPUs. Launching multiple threads within each process lets us scale even
further without incurring the overhead of more processes.

Pull Request Pull Request #138: Use processes and threads in Azure cloud plugins

Run Details

414 of 466 branches covered (88.84%)

Branch coverage included in aggregate %.

280 of 280 new or added lines in 8 files covered. (100.0%)

1946 of 2160 relevant lines covered (90.09%)

2.7 hits per line