grobidOrg / grobid / 28442091951

39%

push

github

Make tokenizer language-sensitive in PDFALTOSaxHandler (#1480)

* Initial plan

* Make tokenizer language-sensitive in PDFALTOSaxHandler

- Add Language field initialized from doc.getLanguage() in constructor
- Add setLanguage()/getLanguage() methods for external configuration
- Pass language to analyzer.tokenize() call instead of using default
- Add tests for language initialization and configuration
- Fixes #376

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

8677 of 24982 branches covered (34.73%)

Branch coverage included in aggregate %.

18494 of 45348 relevant lines covered (40.78%)

1.65 hits per line