MITLibraries / hrqb-client / 9957548529

100%

main: 96%

push

github

Support pipeline task includes and excludes

Why these changes are being introduced:

This change was motivated by a desire to more easily control what tasks are
included and excluded when running a pipeline during local development.

Formerly it was possible to simulate selecting a single task from a pipeline to run,
but it really didn't run the pipeline itself in that scenario.  This also made task
output removal finicky, where it was not clear if dealing with a single task or a pipeline.

By introducing more substantive task include and exclude parameters for a pipeline,
that affect both what tasks run and have their data optionally removed, it allows
for more fluid local development.

This shift also suggested some awkward edges that could be reworked, like a
pipeline having its own 'run_pipeline()' method, which allows for some setup
and cleanup.

How this addresses that need:
* utils.luigi.run_pipeline moved to HRQBPipelineTask as a method
* HRQBPipelineTask gets 'included_tasks' and 'excluded_tasks' parameters
  * both are lists of strings, and are checked on pipeline init that they exist
  * explicit includes override the yielded tasks for a pipeline
  * explicit excludes prevent tasks from running and having data cleaned up

Side effects of this change:
* None really.  All effects are for local development.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/HRQB-41

79 of 79 new or added lines in 4 files covered. (100.0%)

963 of 967 relevant lines covered (99.59%)

1.0 hits per line