xolox / python-executor / 155

91%

push

travis-ci

Refactor executor / proc separation of concerns* (backwards incompatible!)

In executor 7.7 the process management functionality was decoupled from
external command execution in order to re-use the process management
functionality in my proc package (this was integrated into proc 0.4).

In retrospect I implemented this refactoring (in November '15) too
hastily because the UNIX signal handling doesn't belong in the
executor package (it's meant to be portable).

Last weekend I decided to finally do something about this! I'm only
committing this now because it took me days to clean up, stabilize,
document and test the refactoring :-). A high level summary:

 - All process manipulation that uses UNIX signals is being moved to
   the `proc' package, that includes things like SIGSTOP / SIGCONT.
   This means that the methods ControllableProcess.suspend() and
   ControllableProcess.resume() are no longer available.

   This will break fresh installations of my `proc' package until I
   release a new version, because I haven't pinned the max version of
   dependencies I control. The new release of `proc' is waiting to be
   uploaded though :-).

 - The `executor' package no longer keeps references to subprocess.Popen
   objects after the process has finished, to allow garbage collection.
   This should resolve an issue I was seeing recently when I was pushing
   the limits of executor command pools and ran into `IOError: [Errno 24]
   Too many open files'.

   Someone on StackOverflow with the same problem:
   http://stackoverflow.com/questions/6669996/python-subprocess-running-out-of-file-descriptors

   Someone on StackOverflow who knows how to fix it:
   http://stackoverflow.com/a/23763193/788200

   While implementing this refactoring I had a lot of trouble making
   sure that ExternalCommand.pid and ExternalCommand.returncode would be
   preserved when the ExternalCommand.subprocess reference was destroyed
   (it seems so obvious, but nevertheless this t... (continued)

142 of 142 new or added lines in 3 files covered. (100.0%)

942 of 1013 relevant lines covered (92.99%)

4.64 hits per line