kibertoad / gmx-word-counter / 28665100474

100%

push

github

Handle Tibetan-script and Arabic-script languages explicitly (#131)

Two gaps in language coverage where the generic word-splitting regex
produces a wrong count:

Tibetan (bo) and Dzongkha (dz) are written in the Tibetan script, which
delimits syllables with the tsheg (U+0F0B) rather than spaces between
words. The generic regex counted every syllable as a separate word. GMX-V
2.0 defines no word-count factor for them and there is no dictionary-based
segmentation available (ICU ships word dictionaries only for Burmese, CJK,
Khmer, Lao and Thai), so they now return 0, consistent with the existing
handling of Lao, Khmer and Burmese.

Arabic-script languages join letter runs across the zero-width non-joiner
(U+200C), zero-width joiner (U+200D) and tatweel (U+0640). Only Persian was
handled; Urdu, Pashto, Sindhi, Central Kurdish and Uyghur fell through to
the generic regex, which split ZWNJ-joined words and overcounted. The
Persian regex is matched on \p{Script=Arabic}, so the same pattern is now
shared across all of these (and Arabic itself, whose count is unchanged).

Also adds a test for the empty-language-subtag fallback, bringing branch
coverage back to 100%.

90 of 90 branches covered (100.0%)

Branch coverage included in aggregate %.

71 of 71 relevant lines covered (100.0%)

204.23 hits per line