MITLibraries / transmogrifier
99%
main: 99%

LAST BUILD BRANCH: v2.6
DEFAULT BRANCH: main
Repo Added
Files 19
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH TIMX-403-inputs-support-parquet-writing
branch: TIMX-403-inputs-support-parquet-writing
CHANGE BRANCH
x

coverage: 99.464% (+0.004%) from 99.46%
11921190435

push

github

ghukill 
Add run_id CLI argument and Transformer attribute

Why these changes are being introduced:

As we move into Transmogrifier writing to a parquet dataset, one
important bit of information it will need is the concept of a
"run id".  This correlates directly to an "Execution UUID" that
every StepFunction invocation produces.  This identifier is then
used when writing the records to the parquet dataset, allowing
for quick and easy access to records associated with that identifier.

There is a small many-to-one relationship that makes naming a bit
awkward: each StepFunction invocation may run Transmogrifier multiple
times (e.g. multiple input files).  Each time it invokes Transmogrifier,
the same "run_id" would be passed.  This effectively groups the outputs
of all Transmogrifier invocations in the same location in the parquet
dataset.  The language of this new "run_id" in Transmogrifier is
intentionally somewhat high level, indicating it's just an identifier
to associate with that invocation of the run.

How this addresses that need:
* Adds new CLI argument -r / --run-id
* Transformer gets new attribute 'run_id'
* Transformer mints a UUID of a run id is not passed, making
the change backwards compatible and inconsequential if a run
id is not passed

Side effects of this change:
* Going forward, invocations of Transmogrifier can use the run id
as part of the parquet record writing.  Until then, it has no effect.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-403

13 of 13 new or added lines in 2 files covered. (100.0%)

1669 of 1678 relevant lines covered (99.46%)

0.99 hits per line

Relevant lines Covered
1678 RELEVANT LINES 1669 COVERED LINES
0.99 HITS PER LINE
Source Files on TIMX-403-inputs-support-parquet-writing

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
11921190435 TIMX-403-inputs-support-parquet-writing Add run_id CLI argument and Transformer attribute Why these changes are being introduced: As we move into Transmogrifier writing to a parquet dataset, one important bit of information it will need is the concept of a "run id". This correlates d... push ghukill github
99.46
See All Builds (691)