kobotoolbox / kpi / 27639173196 / 8

81%

master: 76%

push

github

feat(nlp): ensure maximum language support with per-language STT routing and GLOBAL/EU toggle DEV-2277 (#7154)

### 📣 Summary
Extends the Google ASR/MT location handling from
[kpi#7128](https://github.com/kobotoolbox/kpi/pull/7128) to support
every language Google offers. Instead of a fixed `us`/`eu` endpoint for
all languages, the code now routes each language to whichever Google
endpoint hosts its best available model. A new `GLOBAL`/`EU` Constance
toggle replaces the old `US`/`EU` toggle: `GLOBAL` (default) uses
per-language routing for maximum coverage; `EU` restricts all processing
to EU-hosted endpoints for data residency compliance.

### 📖 Description
[kpi#7128](https://github.com/kobotoolbox/kpi/pull/7128) re-implemented
the server-wide Google ASR/MT region setting (`ASR_MT_GOOGLE_REGION`)
and simplified routing to use a single fixed location: `us` for all
languages on non-EU servers and `eu` on EU servers.

However, this caused 24 languages to lose ASR support, languages like
Irish, Belarusian, Bosnian, and Somali that only have `chirp_2` model
support in specific Google sub-regions (`us-central1`, `europe-west4`)
and no model at all in the `us`/`eu` multi-region endpoints.

Because we cannot lose support for these 24 languages, we decided to
implement the following approach instead:
1. Global mode (default): Support every language that Google supports by
routing each language to the endpoint that provides the best available
model.
2. EU mode: Use the best available model within EU-hosted endpoints
only. Languages that are not available in any EU endpoint are considered
unsupported on EU servers.

In `GLOBAL` mode, the code reads `location_code` per language from the
database (populated by the language spreadsheet, which stores the
correct global location for each model). In EU mode, the code overrides
the DB location at runtime using the table above via
`EU_LOCATION_BY_MODEL` in `locations.py`.
---------

Co-authored-by: Olivier Léger <olivi... (continued)

5945 of 12399 branches covered (47.95%)

22501 of 37181 relevant lines covered (60.52%)

0.61 hits per line