cloudmarker / cloudmarker / 855

84%

push

travis-ci-com

Use processes and threads in Azure cloud plugins

Both `AzCloud` and `AzVM` now rely on multiprocessing and multithreading
to pull resources concurrently from Azure cloud in order to complete its
scan faster.

All of the multiprocessing and multithreading code involved is put in a
new `ioworkers` module. `AzCloud` and `AzVM` invoke `ioworkers.run()` to
launch multiple worker processes each of which launch multiple worker
threads. The callers, i.e., `AzCloud` and `AzVM`, need to provide this
`ioworkers.run()` function two callback functions:

  - An input function which when invoked yields tuples where each tuple
    defines a unit of work.
  - An output function which works on each tuple yielded by the input
    function, does something with it, and then yields outputs.

The output functions are run concurrently by `ioworkers` in multiple
worker threads within multiple worker processes. This two-level
approach, i.e., worker threads within worker processes, has been chosen
to maximize utilization of CPUs. This approaches provides a good balance
between the following two extremes:

  - Using a large number of processes only would consume too many IPC
    resources and increase the overhead of context switching for
    processes.
  - Using a large number of threads only does not scale well in Python
    due to global interpreter lock (GIL) which allows only one thread to
    execute at a time.

Lauching multiple workers sidesteps GIL and lets us utilize multiple
CPUs. Launching multiple threads within each process lets us scale even
further without incurring the overhead of more processes.

414 of 466 branches covered (88.84%)

Branch coverage included in aggregate %.

1946 of 2160 relevant lines covered (90.09%)

2.7 hits per line