MITLibraries / transmogrifier / 21484421274

99%

main: 99%

Pull #268

github

More targeted full-text extraction for mitlibwebsite

Why these changes are being introduced:

It was decided that the full-text getting extracted from mitlibwebsite full HTML
was too broad.  We were collecting header and footer data that was not unique to
the record/URL at hand.

How this addresses that need:

After some analysis by DiscoEng, some URL + element selector patterns were identified
to target meaningful container elements.  This has dramatically reduced the amount of full-text
while increasing the quality at the same time.

Side effects of this change:
* mitlibwebsite TIMDEX records have higher quality fulltext field values

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/USE-364

25 of 25 new or added lines in 1 file covered. (100.0%)

1827 of 1847 relevant lines covered (98.92%)

0.99 hits per line