1
96%
main: 86%

Ran 29 Oct 2024 05:17PM UTC

Files 14

Run time 1s

Badge

Embed ▾

Committed 29 Oct 2024 05:13PM UTC coverage: 96.486% (+0.1%) from 96.386%

Job # 11579093636.1

Build Type

push

github

Committed by

ghukill

Commit Message

Dedupe records as part of collate_ab_transforms

Why these changes are being introduced:

Ideally, we could provide files to ABDiff in the way they
accumulate in S3 where full runs are followed by dailes,
sometimes there are deletes, etc.  But in doing so, we would
only end up with the most recent, non-deleted version of
the record in the final dataset to analyze.

How this addresses that need:
* Adds step in collate_ab_transforms to dedupe records
based on timdex_record_id, run_date, run_type, and action

Side effects of this change:
* Analysis will only contain the most recent version of a record
even if the record is duplicated amongst input files.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-371

Run Details

659 of 683 relevant lines covered (96.49%)

0.96 hits per line

MITLibraries / transmogrifier-ab-diff / 11579093636 / 1
96%
main: 86%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 11579093636.1

MITLibraries / transmogrifier-ab-diff / 11579093636 / 1 96% main: 86%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 11579093636.1

MITLibraries / transmogrifier-ab-diff / 11579093636 / 1
96%
main: 86%

README BADGES
x