Freegle / Iznik / 27630

72%

master: 72%

Pull #1105

circleci

fix(moderation): replace trigram language detector with ELD to stop false "foreign language" flags

The content-check language filter used patrickschur/language-detection, a
character-trigram model that scores character-frequency profiles, not words. On
short, list-style Freegle posts (item / condition / collection lines) it mis-ranked
plain English as a Latinate language and flagged it as foreign (Discourse #9919: an
all-English-words post flagged as French), while genuine foreign posts sometimes
ranked English close behind and slipped through. Tuning the accept ratio (#1103,
0.9->0.8) only slides that see-saw, and demonstrably lets more foreign spam through.

Swap both detection paths (ContentCheckService + Mail/Incoming/SpamCheckService) to
nitotm/efficient-language-detector (ELD): it picks the correct top language on short
text and exposes isReliable(), so we flag only when confident the text is neither
English nor Welsh, and leave ambiguous short text alone. Native PHP, no sidecar.
Empirically 0 false-positive / 0 false-negative on the failing cases.

Supersedes #1103.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CQZ56NwFYEuhrtaoU7oYoM

12608 of 14554 branches covered (86.63%)

Branch coverage included in aggregate %.

18 of 19 new or added lines in 2 files covered. (94.74%)

137835 of 192507 relevant lines covered (71.6%)

37.48 hits per line