MITLibraries / transmogrifier / 20113036624

99%

main: 99%

Pull #266

github

Parse full HTML from mitlibwebsite source records

Why these changes are being introduced:

Now that browsertrix-harvester is including full HTML + response headers in the
source record available to Transmogrifier, we can do two things:

1. Parse metadata for mitlibwebsite TIMDEX records from the original, full HTML
in a more opinionated fashion than we could in browsertrix-harvester.

2. Extract good, meaningful full-text from the full HTML to use for the new
`fulltext` field.

How this addresses that need:

Expects a new `html_base64` field in the browsertrix-harvester source records.
Uses this to extract metadata and full-text for the record.

Side effects of this change:
* Full-text is now available in the TIMDEX record for the mitlibwebsite
source.
* If needed, this HTML parsing could be utilized to extract more granular,
source specific metadata in the future.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/USE-259

23 of 24 new or added lines in 1 file covered. (95.83%)

1795 of 1817 relevant lines covered (98.79%)

0.99 hits per line