MITLibraries / timdex-pipeline-lambdas / 20312802350

96%

main: 96%

push

github

Further utilize and control next-step values

Why these changes are being introduced:

This lambda project has a tightly coupled relationship with the TIMDEX ETL StepFunction.
Currently, at three points in the StepFunction this lambda is invovked to craft commands
for the next application in the workflow.  What's not as well established is how this
lambda is also providing flow control values, i.e. when to exit the StepFunction early.

With the shift to the TIMDEX dataset, additional harvesters, and soon new parts of the
workflow, it's becoming increasingly important that the lambdas help guide the StepFunction
for when to exit early, what is accepted vs error behavior, etc.

How this addresses that need:

Building on the recent refactor, we can begin to lean on the `next-step` values a
bit more in the StepFunction.  Formerly, it was just used to "jump" to extract,
transform, or load work.  But by adding values like `exit-ok`, `exit-error`, and
`end`, Choice states in the StepFunction can utilize these for flow control based on
information the lambda has determined.

Specifically for this ticket, we now set `next-step` values pre-transform and pre-load
steps that inform the StepFunction to exit early if needed.  This will require updated
Choice states in the StepFunction to key off these new `next-step` values.

Side effects of this change:
* StepFunction will operate as before without changes.
* When Choice states are added and/or updated, the StepFunction will react to new
`next-step` values emitted by the lambda

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/USE-279

18 of 18 new or added lines in 1 file covered. (100.0%)

351 of 365 relevant lines covered (96.16%)

0.96 hits per line