MITLibraries / hrqb-client / 9957548529 / 1
100%
main: 96%

LAST BUILD BRANCH: v1.14.0
DEFAULT BRANCH: main
Ran
Files 26
Run time 1s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 99.586%. First build
9957548529.1

push

github

ghukill 
Support pipeline task includes and excludes

Why these changes are being introduced:

This change was motivated by a desire to more easily control what tasks are
included and excluded when running a pipeline during local development.

Formerly it was possible to simulate selecting a single task from a pipeline to run,
but it really didn't run the pipeline itself in that scenario.  This also made task
output removal finicky, where it was not clear if dealing with a single task or a pipeline.

By introducing more substantive task include and exclude parameters for a pipeline,
that affect both what tasks run and have their data optionally removed, it allows
for more fluid local development.

This shift also suggested some awkward edges that could be reworked, like a
pipeline having its own 'run_pipeline()' method, which allows for some setup
and cleanup.

How this addresses that need:
* utils.luigi.run_pipeline moved to HRQBPipelineTask as a method
* HRQBPipelineTask gets 'included_tasks' and 'excluded_tasks' parameters
  * both are lists of strings, and are checked on pipeline init that they exist
  * explicit includes override the yielded tasks for a pipeline
  * explicit excludes prevent tasks from running and having data cleaned up

Side effects of this change:
* None really.  All effects are for local development.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/HRQB-41

963 of 967 relevant lines covered (99.59%)

1.0 hits per line

Source Files on job 9957548529.1