MITLibraries / transmogrifier-ab-diff / 11350173928

86%

push

github

Add 'collate_ab_transforms' command (#22)

* Add 'collate_ab_transforms' command

Why these changes are being introduced:
* The A/B diff workflow requires a function that can
combine the transformed records (stored in JSON files)
generated by A/B versions of transmogrifier into a
single data structure that can be used for analysis
downstream. It was decided for this function to
create a PyArrow dataset and create Parquet file(s)
that contain joined records from A/B transformed files.

How this addresses that need:
* Create function that reads transformed files and yields
  dictionaries describing the transformed records.
* Create generator that yields pyarrow.RecordBatch(es) of
  TIMDEX record dictionaries
* Write TIMDEX record dictionaries to a temporary Parquet
  dataset for joining.
* Create function that uses DuckDB to query the temporary
  Parquet dataset, joins A/B versions of records, and
  yields query result in pyarrow.RecordBatch(es).
* Write batches of joined records to a Parquet dataset
  ("collated") in the run directory.
* Create function that parses version and filename from transformed file.
* Add util for writing Parquet datasets.

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-349

60 of 61 new or added lines in 3 files covered. (98.36%)

319 of 324 relevant lines covered (98.46%)

0.98 hits per line