MITLibraries / timdex-pipeline-lambdas / 12604262226

97%

main: 96%

push

github

Update load commands for v2 parquet dataset

Why these changes are being introduced:

The pipeline lambdas will need to produce slightly different load
commands for timdex-index-manager (TIM) for the v2 parquet dataset.

The signature is different enough for generating the load commands
that feaure flagging branching is slightly different than the
transform command branching.

How this addresses that need:
* Updates generate_load_commands() to produce v2 output, invoking
a new TIM CLI command bulk-update.
* Add a generate_load_commands_v1() function that will continue
to support v1 invocations, but will be removed after feature flag
work is complete.
* Installs the timdex-dataset-api (TDA) library for querying the
TIMDEX dataset and looking for records for a given run.

Side effects of this change:
* The pipeline lambdas should remain backwards v1 compatible, but
will now support v2 runs.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-442

37 of 42 new or added lines in 3 files covered. (88.1%)

310 of 318 relevant lines covered (97.48%)

0.97 hits per line