MITLibraries / transmogrifier-ab-diff / 11579093636

96%

main: 86%

push

github

Dedupe records as part of collate_ab_transforms

Why these changes are being introduced:

Ideally, we could provide files to ABDiff in the way they
accumulate in S3 where full runs are followed by dailes,
sometimes there are deletes, etc.  But in doing so, we would
only end up with the most recent, non-deleted version of
the record in the final dataset to analyze.

How this addresses that need:
* Adds step in collate_ab_transforms to dedupe records
based on timdex_record_id, run_date, run_type, and action

Side effects of this change:
* Analysis will only contain the most recent version of a record
even if the record is duplicated amongst input files.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-371

26 of 26 new or added lines in 1 file covered. (100.0%)

659 of 683 relevant lines covered (96.49%)

0.96 hits per line