PyCQA / pylint / 3546

90%

main: 95%

Pull #3016

travis-ci

Refactor file checking for a simpler parallel API

This change prepares the code for enabling Prospector to take advantage
of running PyLint parallel.

Iterating files is moved into generator (_iterate_file_descrs) so that
parallel checking can use the same implementation (_check_file) just
by providing different kind of generator that reads the files from parent
process.

The refactoring removes code duplication that existed in PyLinter._do_check
method; checking module content from stdin had identical implementation to
checking content from a source file.

Made PyLinter.expand_files a private method.

The previous implementation of parallel linting created new PyLinter
objects in the worker (child) process causing failure when running under
Prospector because Prospector uses a custom PyLinter class
(a class inherited from PyLinter)
and PyLint naturally just creates PyLinter object. This caused linting to
fail because there is options for Prospector's IndentChecker which was not
created in the worker process.

The new implementation passes the original PyLinter object into workers
when the workers are created. See https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Note that as Windows uses spawn method by default, PyLinter object (and
its) members need to be pickleable from now on with the exception being
PyLinter.reporter which is not passed to child processes.

The performance has remained about the same based on quick tests done with
Django project containing about 30 000 lines of code; with the old
implementation linting took 26-28 seconds with 8 jobs on quad core i7 and
24-27 seconds with the new implementation.

100 of 100 new or added lines in 4 files covered. (100.0%)

32225 of 35871 relevant lines covered (89.84%)

4.48 hits per line