MITLibraries / transmogrifier / 10355221461

99%

main: 99%

Pull #206

github

Deduplicate list fields in TIMDEX record

Why these changes are being introduced:
* Improve data quality of TIMDEX records
by reducing duplication of data in list fields.

How this addresses that need:
* Create an attrs converter function to dedupe list of items
* Create ListFields abstract class with hash method
* Set hash methods in custom classes to ListFields.__hash__
* Set 'converter=dedupe' for every list field in TimdexRecord
* Add unit tests verifying deduplication of list fields

Side effects of this change:
* Deduplication is highly likely to result in diffs
when comparing transformed records before and after this
change. However (and more importantly), reducing duplicates
improves the data quality of TIMDEX records.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-332

31 of 32 new or added lines in 1 file covered. (96.88%)

1674 of 1683 relevant lines covered (99.47%)

2.98 hits per line