kp-forks / grobid / 28001811224

push

github

fix: stop folding the letters ae/oe to "ae"/"oe" in clean() #728 (#1461)

TextUtilities.clean() expands typographic ligatures (the fi/fl/ff family,
genuine PDF rendering artifacts) but also folded the distinct Unicode
letters ae/AE and oe/OE to ASCII. Those are real letters used deliberately
in Danish, Norwegian, Icelandic, French, etc., so folding them was data
loss in the extracted text. They now pass through unchanged; the ligature
expansion is unaffected.

Add regression tests covering both the preserved letters and the
still-expanded typographic ligatures.

8635 of 24856 branches covered (34.74%)

Branch coverage included in aggregate %.

18384 of 45184 relevant lines covered (40.69%)

1.65 hits per line