kobotoolbox / kpi / 29437572618
81%
master: 76%

LAST BUILD BRANCH: restore-deployment-data-query-ability
DEFAULT BRANCH: master
Ran
Jobs 10
Files 894
Run time 3min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 81.373% (-0.02%) from 81.393%
29437572618

push

github

web-flow 
fix(nlp): add configurable Google STT language code overrides and improve batch error handling DEV-2403 (#7248)

### 📣 Summary
Adds configurable language code overrides for Google Speech-to-Text v2
(e.g. mapping Swahili to `auto`) and improves batch transcription error
handling by surfacing per-file Google API errors instead of generic
missing-output failures.

### 📖 Description
#### Problem

Google Speech-to-Text v2 currently fails to transcribe Swahili when
using the standard language codes (`sw` or `sw-KE`), even though the
language is supported. However, using `auto` allows Google to correctly
detect and transcribe the audio.

Additionally, batch transcription failures could sometimes be reported
as "No transcription output written to GCS" even though Google had
already returned a more specific per-file error.

#### Solution

This PR introduces configurable language code overrides for Google
Speech-to-Text and improves error handling for batch transcription
responses.

#### Changes

- Added a configurable `ASR_LANGUAGE_CODE_OVERRIDES` Constance setting
with environment variable support.
- Added parsing for comma-separated language mappings (e.g.
`sw:auto,sw-KE:auto`).
- Applied language overrides only within the Google Speech-to-Text
integration before sending requests to Google, keeping Kobo language
definitions unchanged.
- Added informational logging whenever a configured language override is
applied.
- Improved Speech-to-Text v2 batch error handling by detecting per-file
errors returned by Google before attempting to read output files from
Cloud Storage.
- Added clearer logging and user-facing error messages for per-file
transcription failures.

### 👷 Description for instance maintainers

This PR introduces a default language-code override for Google
Speech-to-Text:
- sw -> auto
- sw-KE -> auto

This workaround is enabled by default because Google Speech-to-Text v2
currently fails to transcribe Swahili when explicitly using `sw` or
`sw-KE`, wh... (continued)

9298 of 12651 branches covered (73.5%)

3 of 55 new or added lines in 2 files covered. (5.45%)

4382 existing lines in 157 files now uncovered.

30292 of 37226 relevant lines covered (81.37%)

5.51 hits per line

Uncovered Changes

Lines Coverage ∆ File
35
17.87
 -25.93% kobo/apps/subsequences/integrations/google/google_transcribe.py
17
36.36
 -40.56% kobo/apps/subsequences/integrations/google/locations.py

Coverage Regressions

Lines Coverage ∆ File
211
26.4
 -59.27% kpi/serializers/v2/asset_permission_assignment.py
185
14.34
 -66.31% kobo/apps/subsequences/tasks.py
165
30.61
 -56.12% kobo/apps/subsequences/models.py
150
40.73
 -54.55% kobo/apps/organizations/serializers.py
150
28.85
 -57.69% kobo/apps/subsequences/serializers.py
142
37.13
 -46.25% kpi/views/v2/data.py
126
31.44
 -64.95% kobo/apps/subsequences/actions/automatic_bedrock_qual.py
126
19.9
 -64.29% kobo/apps/subsequences/utils/versioning.py
123
35.57
 -48.62% kobo/apps/subsequences/actions/base.py
119
64.26
 -18.65% kpi/serializers/v2/asset.py
107
15.7
 -47.98% kobo/apps/superuser_stats/tasks.py
105
47.45
 -17.3% kpi/deployment_backends/openrosa_backend.py
91
19.51
 -73.98% kobo/apps/subsequences/actions/qual.py
86
26.12
 -64.18% kobo/apps/trash_bin/utils/trash.py
84
17.76
 -78.5% kobo/apps/subsequences/actions/mixins.py
81
0.0
 -93.1% kobo/apps/stripe/serializers.py
75
0.0
 -41.9% kobo/apps/stripe/views.py
71
20.58
 -29.22% kobo/apps/subsequences/integrations/google/google_translate.py
70
37.21
 -40.7% kpi/serializers/current_user.py
69
57.23
 -41.57% kobo/apps/organizations/views.py
68
41.03
 -43.59% kobo/apps/organizations/admin/organization_user.py
68
27.2
 -54.4% kobo/apps/stripe/utils/subscription_limits.py
64
17.87
 -25.93% kobo/apps/subsequences/integrations/google/google_transcribe.py
63
0.0
 -51.64% kobo/apps/stripe/models.py
62
27.87
 -50.82% kobo/apps/project_ownership/tasks.py
60
34.67
 -40.0% kobo/apps/project_ownership/utils.py
60
26.85
 -55.56% kpi/serializers/v2/paired_data.py
58
29.29
 -41.43% kpi/serializers/v2/asset_file.py
55
31.62
 -47.01% kpi/serializers/v2/export_task.py
52
42.86
 -49.52% kpi/views/v2/environment.py
49
20.21
 -52.13% kobo/apps/stripe/utils/billing_dates.py
44
52.58
 -45.36% kobo/apps/subsequences/views.py
43
31.43
 -40.95% kobo/apps/organizations/admin/organization.py
42
0.0
 -79.25% kobo/apps/user_reports/tasks.py
41
32.81
 -64.06% kobo/apps/trash_bin/utils/account.py
40
61.21
 -24.24% kobo/apps/organizations/models.py
40
61.89
 -10.81% kpi/views/v2/asset.py
38
37.14
 -54.29% kobo/apps/organizations/permissions.py
38
49.47
 -40.0% kpi/views/v2/asset_permission_assignment.py
35
77.61
 -17.41% kobo/apps/project_ownership/models/transfer.py
32
66.91
 -7.9% kpi/deployment_backends/base_backend.py
31
26.42
 -58.49% kobo/apps/trash_bin/utils/project.py
29
55.36
 -25.89% kobo/apps/openrosa/apps/logger/utils/instance.py
26
46.03
 -20.63% kobo/apps/project_views/views.py
25
31.11
 -55.56% kpi/serializers/v2/data.py
24
59.72
 -33.33% kobo/apps/project_ownership/serializers/invite.py
24
27.63
 -31.58% kobo/apps/subsequences/integrations/google/rate_limit.py
23
27.4
 -31.51% kobo/apps/trackers/utils.py
22
56.81
 -10.33% kpi/filters.py
21
0.0
 -84.0% kobo/apps/stripe/signals.py
21
48.84
 -48.84% kobo/apps/trash_bin/models/project.py
20
71.43
 -23.81% kpi/serializers/v2/service_usage.py
20
43.7
 -14.81% kpi/views/v2/asset_snapshot.py
19
26.0
 -38.0% kobo/apps/service_health/views.py
19
0.0
 -65.52% kobo/apps/stripe/admin.py
19
47.73
 -43.18% kobo/apps/trash_bin/models/attachment.py
19
51.28
 -48.72% kobo/apps/trash_bin/tasks/__init__.py
18
66.4
 -14.4% kpi/models/paired_data.py
16
32.0
 -64.0% kobo/apps/openrosa/apps/logger/utils/attachment.py
16
0.0
 -100.0% kobo/apps/service_health/test_service_health.py
16
59.32
 -27.12% kpi/views/v2/export_task.py
16
34.55
 -14.55% kpi/views/v2/paired_data.py
15
28.57
 -71.43% kobo/apps/trash_bin/utils/signals.py
15
69.23
 -19.23% kpi/authentication.py
15
55.81
 -34.88% kpi/backends.py
15
40.48
 -35.71% kpi/serializers/v2/deployment.py
14
31.82
 -31.82% kobo/apps/stripe/utils/limit_enforcement.py
14
41.3
 -15.22% kobo/apps/subsequences/integrations/google/base.py
14
75.94
 -10.53% kpi/paginators.py
13
66.32
 -13.68% kobo/apps/openrosa/libs/utils/middleware.py
13
61.11
 -36.11% kobo/apps/organizations/tasks.py
13
60.53
 -34.21% kobo/apps/organizations/utils.py
13
45.71
 -37.14% kobo/apps/user_reports/seralizers.py
13
74.42
 -6.05% kpi/permissions.py
13
33.33
 -48.15% kpi/utils/prerequisite_migration_checker.py
12
35.0
 -60.0% kobo/apps/stripe/utils/import_management.py
12
43.48
 -52.17% kobo/apps/subsequences/throttling.py
12
57.14
 -42.86% kobo/apps/subsequences/utils/supplement_data.py
12
51.06
 -25.53% kpi/views/v2/import_task.py
11
85.29
 -10.78% kpi/models/asset_file.py
11
70.18
 -19.3% kpi/utils/mailer.py
10
80.28
 -14.08% kobo/apps/project_ownership/models/invite.py
10
39.58
 -20.83% kobo/apps/subsequences/prompts.py
9
0.0
 -29.03% kobo/apps/stripe/utils/manual_subscription.py
9
79.05
 -8.57% kpi/utils/object_permission.py
9
79.17
 -18.75% kpi/views/v2/asset_file.py
8
71.67
 -3.33% kobo/apps/openrosa/apps/viewer/models/parsed_instance.py
8
84.68
 -1.44% kobo/apps/openrosa/libs/utils/logger_tools.py
8
0.0
 -100.0% kobo/apps/service_health/test_service_health_minimal.py
8
0.0
 -100.0% kobo/apps/stripe/urls.py
8
77.78
 -8.08% kpi/password_validation.py
8
85.71
 -11.43% kpi/utils/cache.py
7
0.0
 -100.0% kobo/apps/stripe/apps.py
7
50.0
 -31.82% kobo/apps/trash_bin/mixins/admin.py
7
53.33
 -46.67% kobo/apps/trash_bin/utils/attachment.py
7
75.0
 -6.03% kpi/models/asset_snapshot.py
6
0.0
 -100.0% kobo/apps/stripe/exceptions.py
6
0.0
 -37.5% kobo/apps/stripe/utils/view_utils.py
6
68.42
 -31.58% kobo/apps/subsequences/schemas.py
6
58.46
 -4.62% kobo/apps/trash_bin/admin.py
6
62.96
 -22.22% kobo/apps/trash_bin/models/account.py
6
91.68
 -1.04% kpi/models/asset.py
6
78.29
 -0.99% kpi/models/import_export_task.py
6
61.9
 -14.29% kpi/utils/urls.py
5
36.36
 -40.56% kobo/apps/subsequences/integrations/google/locations.py
5
65.22
 -21.74% kobo/apps/trash_bin/tasks/project.py
5
82.35
 -14.71% kobo/apps/user_reports/views.py
5
92.92
 -1.36% kpi/mixins/object_permission.py
5
77.07
 -2.44% kpi/utils/mongo_helper.py
4
72.22
 -22.22% kobo/apps/subsequences/actions/automatic_google_transcription.py
4
72.22
 -22.22% kobo/apps/subsequences/actions/automatic_google_translation.py
4
33.33
 -14.81% kobo/apps/subsequences/audit.py
4
83.33
 -16.67% kobo/apps/trackers/models.py
4
68.18
 -18.18% kobo/apps/trash_bin/tasks/account.py
4
37.5
 -5.0% kobo/apps/trash_bin/tasks/attachment.py
4
20.0
 -80.0% kpi/utils/placeholders.py
3
90.48
 -2.38% kobo/apps/openrosa/apps/logger/signals.py
3
62.16
 -8.11% kobo/apps/openrosa/apps/logger/utils/database_query.py
3
63.33
 -5.0% kobo/apps/openrosa/apps/main/models/user_profile.py
3
58.06
 -9.68% kobo/apps/organizations/admin/organization_owner.py
3
42.86
 -4.29% kobo/apps/project_ownership/admin.py
3
85.71
 -8.57% kobo/apps/project_views/models/assignment.py
3
92.86
 -7.14% kobo/apps/subsequences/exceptions.py
3
75.0
 -25.0% kobo/apps/superuser_stats/admin.py
3
86.27
 -5.88% kpi/context_processors.py
3
87.61
 -2.65% kpi/exceptions.py
3
70.0
 -30.0% kpi/fields/serializer_method_file.py
3
62.07
 -3.45% kpi/tasks.py
3
78.57
 -10.71% kpi/utils/submission.py
3
30.67
 -4.0% kpi/utils/two_database_configuration_checker.py
2
78.08
 -2.74% kobo/apps/openrosa/libs/utils/image_tools.py
2
42.86
 -28.57% kobo/apps/subsequences/utils/time.py
2
60.26
 -2.56% kpi/deployment_backends/kc_access/utils.py
2
92.73
 -3.64% kpi/signals.py
2
42.17
 -2.41% kpi/utils/storage.py
2
75.0
 -16.67% kpi/views/v2/open_rosa.py
2
88.24
 -11.76% kpi/views/v2/service_usage.py
1
84.27
 -1.12% kobo/apps/reports/report_data.py
1
87.5
 -12.5% kobo/apps/service_health/middleware.py
1
88.89
 -11.11% kobo/apps/subsequences/actions/manual_qual.py
1
92.31
 -7.69% kobo/apps/user_reports/utils/migrations.py
1
70.73
 -2.44% kpi/admin.py
1
96.67
 -3.33% kpi/fields/kpi_uid.py
1
75.0
 -25.0% kpi/fields/read_only.py
1
82.61
 -4.35% kpi/fields/relative_prefix_hyperlinked_related.py
1
80.95
 -4.76% kpi/fields/writable_json.py
1
66.67
 -8.33% kpi/mixins/validation_password_permission.py
1
62.5
 -12.5% kpi/utils/datetime.py
1
93.94
 -1.01% kpi/utils/django_orm_helper.py
1
81.25
 -6.25% kpi/utils/drf_exceptions.py
1
75.0
 -25.0% kpi/utils/gravatar_url.py
1
27.68
 -0.89% kpi/utils/models.py
1
94.12
 -5.88% kpi/utils/strings.py
1
98.06
 -0.97% kpi/utils/usage_calculator.py
1
84.62
 -7.69% kpi/versioning.py
1
55.56
 -2.22% kpi/views/current_user.py
1
42.5
 -2.5% kpi/views/__init__.py
Jobs
ID Job ID Ran Files Coverage
1 29437572618.1 890
49.54
2 29437572618.2 890
44.38
3 29437572618.3 890
52.89
4 29437572618.4 892
54.16
5 29437572618.5 892
45.74
6 29437572618.6 892
66.65
7 29437572618.7 890
58.91
8 29437572618.8 892
50.88
9 29437572618.9 892
60.06
10 29437572618.10 894
68.95
Source Files on build 29437572618