rlworkgroup / garage / 2650

64%

cron

travis-ci-com

Fix sleeping processes (#295)

The joblib package responsible of the MemmappingPool has been updated to
consider any bugs that could produce the sleeping processes in the
parallel sampler. Also the environment variable JOBLIB_START_METHOD has
been removed since it's not implemented by joblib anymore.
However, if run_experiment is interrupted during the optimization steps,
the sleeping processes are still produced. To fix the problem, the child
processes of the parallel sampler ignore SIGINT so they're not killed
while holding a lock that is also acquired by the parent process,
avoiding a dead lock.
The process (thread in TF) used in Plotter is terminated thanks to
registering the method shutdown with function atexit, but one important
step missing was to clean the Queue that interacts with worker process.
The class BatchPolopt has been re-made as InstrumentedBatchPolopt to
notify the test of the different stages in the experiment life cycle so
it can be interrupted with SIGINT.
The test makes sure that the children processes are gone after
the SIGINT is sent, or it throws an assertion error with those processes
that didn't die.
The context manager MasksSignals has been created to handle the
masking of SIGINT in parallel_sampler.
All plotters are appended to a static list, so they can be easily
reachable from run_experiment to call shutdown on them.
In order to check that all processes died after a time out, a loop to
poll for alive processes was implemented.
A warning message for processes that remain after shutdown is printed so
users of garage can reopen the corresponding issue.
A process known as semaphore tracker is spawned from the run_experiment
process, but we cannot stop this process as it ignores SIGINT and
SIGTERM and we haven't access to it. Therefore, it's removed from the
list of children to wait for in both run_experiment and in test_sigint.
Another process that is spawned from run_experiment is the Manager,... (continued)

83 of 83 new or added lines in 14 files covered. (100.0%)

9269 of 14998 relevant lines covered (61.8%)

0.62 hits per line