MITLibraries / transmogrifier-ab-diff / 11630435556

91%

main: 86%

push

github

Generate CSV of input files and use for run-diff

Why these changes are being introduced:

For very large runs, the use of command line, comman seperated input files
for Transmogrifier does not scale.  Two things are needed and helpful:

1. allow CLI command run-diff to accept a list of input files via another method
2. a tool to help craft lists of input files from S3 based on some criteria

How this addresses that need:
* A new CLI command is added 'timdex-sources-csv' that generates a list of input
files based on the S3 bucket set by env var 'TIMDEX_BUCKET'.  This utilizes the
helper methods added via a former commit.  Default is, for all sources, to list
extract files going back to the last full run, then every subsequent daily run.
This also can be limited to a subset of sources.

* Update CLI command run-diff to allow the -i / --input-files argument to
accept a local CSV file which is parsed and used as the input files list.

Side effects of this change:
* None really, except able to more easily support runs with large amounts
of input files.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-379

12 of 21 new or added lines in 2 files covered. (57.14%)

695 of 762 relevant lines covered (91.21%)

0.91 hits per line