MITLibraries / transmogrifier / 12284736785

99%

main: 99%

Build Type

push

github

Committed by ghukill

Commit Message

Write output to parquet dataset

Why these changes are being introduced:

Transmogrifier is getting updated to write its output to a
TIMDEX parquet dataset, instead of standalone JSON and TXT files.
To perform the actual dataset writing, the library timdex-dataset-api
is used.

These changes also make some preliminary updates to the README and CLI
arguments, moving towards parquet dataset writing as the default behavior
for Transmogrifier.  However, "v1" behavior of writing to JSON and TXT
files is still supported, and will be until parquet dataset work is
finalized.

How this addresses that need:
* Installs timdex-dataset-api as application dependency
  * temporarily pinned to v0.2.0, but pinning will get removed
* Uses timdex_dataset_api.TIMDEXDataset class to perform writing
of transformed and source files to dataset
* Changes -o CLI argument alias from --output-file to
--output-location

Side effects of this change:
* None: where ETL_VERSION=1 (default) behavior will remain the same

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-405

Run Details

37 of 45 new or added lines in 2 files covered. (82.22%)

1 existing line in 1 file now uncovered.

1739 of 1761 relevant lines covered (98.75%)

0.99 hits per line