grobidOrg / grobid

39%

master: 39%

LAST BUILD ON BRANCH bugfix/fix-728

branch: bugfix/fix-728

CHANGE BRANCH

x

Reset

Sync Branches

• bugfix/fix-728
• 0.9.0
• bugfix/add-priority-content-type
• bugfix/avoid-IOBE-in-orcid-search
• bugfix/codespell-round-two
• bugfix/correct-git-revision
• bugfix/docker-arm
• bugfix/docker-full-build
• bugfix/fix-149
• bugfix/fix-15
• bugfix/fix-671
• bugfix/fix-849
• bugfix/fix-classification-delft
• bugfix/fix-doi-search
• bugfix/improve-regexes
• bugfix/issue-1024
• bugfix/issue-1355
• bugfix/issue-271
• bugfix/issue-465
• bugfix/issue-512
• bugfix/issue-716
• bugfix/issue-849
• bugfix/issue-920
• bugfix/jep-env-uv
• bugfix/model-selection-flavours
• bugfix/patch-lmdb
• bugfix/reference-segmenter-createTraining
• bugfix/sentence-segmentation-detection
• bugfix/spotless
• bugfix/support-missing-dir
• bugfixes/docker-retag
• bugix/correct-revision
• chore/remove-powermock
• chore/remove-unused-models
• ci/reduce-coveralls
• ci/support-forks
• display-version-in-ui
• feature/add-api-status-in-ui
• feature/add-code-cleaning
• feature/add-ignore-areas-api
• feature/add-pdf-tei-editor
• feature/auto-update-hf-space
• feature/blingfire
• feature/code-formatting
• feature/codeql
• feature/codespell
• feature/coi-ac
• feature/coi-ac-update-light-models
• feature/community-page
• feature/development-api
• feature/docker-summary
• feature/documentation
• feature/extract-any-url-in-fulltext
• feature/fix-bibtex-index
• feature/improve-documentation
• feature/issue-to-documentation
• feature/literature-law-flavour
• feature/literature-law-flavour-eval
• feature/literature-law-flavour-experiment
• feature/literature-law-flavour-experiments
• feature/middle-name-bibtex
• feature/onnx-models
• feature/onnx-models-perf
• feature/onnx-models-perfs
• feature/otlp-metrics-push
• feature/prepare-release
• feature/refactor-tei-formatter
• feature/refactor-trainers
• feature/remove-unused-lexicons
• feature/rename-ui-items
• feature/revise-consolidation
• feature/rewrite-overloaded
• feature/sync-crf-manual-build
• feature/unpin-lmdb-delft
• feature/update-benchmarks
• feature/update-copyright
• feature/update-delft
• feature/update-dependencies
• feature/update-doc-flavor-dev
• feature/update-editorconfig
• feature/update-lingua
• feture/update-documentation
• fix/pdfalto-memory-limit-macos
• hotfix/0.9.0-docker
• improvement/affiliation-author
• improvement/fix-lexicon
• improvements/fails-on-failures
• improvements/speed-up-ci
• master
• multi-arch-docker-image
• publish/0.9.0-central
• release/0.9.0
• rootless
• security/cmdinjection-threadleak
• security/jline-cmdinjection-threadleak
• security/update-dependencies
• tensorflow-2.16
• tensorflow-2.16-docker-full

Build # 27987196011

Build Type

Pull #1461

github

Committed by lfoppiano

Commit Message

fix: stop folding the letters ae/oe to "ae"/"oe" in clean() #728

TextUtilities.clean() expands typographic ligatures (the fi/fl/ff family,
genuine PDF rendering artifacts) but also folded the distinct Unicode
letters ae/AE and oe/OE to ASCII. Those are real letters used deliberately
in Danish, Norwegian, Icelandic, French, etc., so folding them was data
loss in the extracted text. They now pass through unchanged; the ligature
expansion is unaffected.

Add regression tests covering both the preserved letters and the
still-expanded typographic ligatures.

Pull Request Pull Request #1461: Fix 728 - Avoid converting danish letters when not necessary

Coverage Stats

8636 of 24856 branches covered (34.74%)

Branch coverage included in aggregate %.

18390 of 45184 relevant lines covered (40.7%)

4.94 hits per line