kobotoolbox / kpi / 27622545666

81%

master: 76%

Build Type

push

github

Committed by web-flow

Commit Message

feat(nlp): ensure maximum language support with per-language STT routing and GLOBAL/EU toggle DEV-2277 (#7154)

### 📣 Summary
Extends the Google ASR/MT location handling from
[kpi#7128](https://github.com/kobotoolbox/kpi/pull/7128) to support
every language Google offers. Instead of a fixed `us`/`eu` endpoint for
all languages, the code now routes each language to whichever Google
endpoint hosts its best available model. A new `GLOBAL`/`EU` Constance
toggle replaces the old `US`/`EU` toggle: `GLOBAL` (default) uses
per-language routing for maximum coverage; `EU` restricts all processing
to EU-hosted endpoints for data residency compliance.

### 📖 Description
[kpi#7128](https://github.com/kobotoolbox/kpi/pull/7128) re-implemented
the server-wide Google ASR/MT region setting (`ASR_MT_GOOGLE_REGION`)
and simplified routing to use a single fixed location: `us` for all
languages on non-EU servers and `eu` on EU servers.

However, this caused 24 languages to lose ASR support, languages like
Irish, Belarusian, Bosnian, and Somali that only have `chirp_2` model
support in specific Google sub-regions (`us-central1`, `europe-west4`)
and no model at all in the `us`/`eu` multi-region endpoints.

Because we cannot lose support for these 24 languages, we decided to
implement the following approach instead:
1. Global mode (default): Support every language that Google supports by
routing each language to the endpoint that provides the best available
model.
2. EU mode: Use the best available model within EU-hosted endpoints
only. Languages that are not available in any EU endpoint are considered
unsupported on EU servers.

In `GLOBAL` mode, the code reads `location_code` per language from the
database (populated by the language spreadsheet, which stores the
correct global location for each model). In EU mode, the code overrides
the DB location at runtime using the table above via
`EU_LOCATION_BY_MODEL` in `locations.py`.
---------

Co-authored-by: Olivier Léger <olivi... (continued)

Coverage Stats

9246 of 12487 branches covered (74.05%)

9 of 26 new or added lines in 2 files covered. (34.62%)

2 existing lines in 1 file now uncovered.

30552 of 37264 relevant lines covered (81.99%)

5.72 hits per line