ocadotechnology / bbq

88%

master: 85%

LAST BUILD ON BRANCH DatastoreTableGetRetriableException500error_less_likely_to_happen

branch: DatastoreTableGetRetriableException500error_less_likely_to_happen

CHANGE BRANCH

x

Reset

• DatastoreTableGetRetriableException500error_less_likely_to_happen
• A_MESSAGE_CORRECTION
• CONFIGURATION_UNIFICATION
• COPY_JOB_DEADLINE_EXCEEDED
• NON-JIRA_updating_orphaned_backup_query
• NON_JIRA_do_backup_when_size_mismatch
• SLI_DATASTUDIO_BUG
• SLI_documentation
• SLI_history_view_shows_rows_only_from_newest_snapshot
• YACHT-1003
• YACHT-1005
• YACHT-1007-contribution-guide
• YACHT-1008_added_google_group
• YACHT-1009_copy_job_should_handle_duplicated_job_id
• YACHT-1012_slo_service_should_handle_tables_without_schema
• YACHT-1020
• YACHT-1021
• YACHT-1023
• YACHT-1025
• YACHT-1026
• YACHT-1027
• YACHT-1029
• YACHT-1032_TF_update
• YACHT-1032_added_datasets_ACLs_to_TF_config
• YACHT-1052
• YACHT-1054
• YACHT-1057
• YACHT-1059
• YACHT-1092
• YACHT-1093
• YACHT-1093-3
• YACHT-1094
• YACHT-1095
• YACHT-1098
• YACHT-1099
• YACHT-1104
• YACHT-1106
• YACHT-1106-2
• YACHT-1111
• YACHT-1123_unify_UI
• YACHT-1124
• YACHT-1124-2
• YACHT-1127
• YACHT-1133
• YACHT-1133-REVERT
• YACHT-1142
• YACHT-1144
• YACHT-1173
• YACHT-1176
• YACHT-1177
• YACHT-1183_IAM_roles_in_TF_config
• YACHT-1183_IAM_update
• YACHT-1184
• YACHT-1185
• YACHT-1186
• YACHT-1187_gae_firewall
• YACHT-1232
• YACHT-1246
• YACHT-1248
• YACHT-1250
• YACHT-1251
• YACHT-1254
• YACHT-1254_export_fix
• YACHT-1256
• YACHT-1256_NEW
• YACHT-1274
• YACHT-1295_document_not_supported_use_cases
• YACHT-1312
• YACHT-1312_MR
• YACHT-1316
• YACHT-1333_update_jinja2_version
• YACHT-1351
• YACHT-1443
• YACHT-361
• YACHT-899
• YACHT-903
• YACHT-910
• YACHT-917
• YACHT-920
• YACHT-962
• YACHT-963
• YACHT-964
• YACHT-965
• YACHT-972
• YACHT-985
• YACHT-986-update-documentation
• YACHT_1001
• YACHT_1002
• YACHT_1002_2
• YACHT_1014
• YACHT_1014_
• YACHT_1051
• YACHT_1068
• YACHT_1301
• YACHT_1301_2
• YACHT_1317
• YACHT_1355
• YACHT_753
• YACHT_905
• YACHT_948
• YACHT_997
• backup_entity_schema_migration_tool
• bug_in_SLI_query
• cleanup_not_used_datastore_entities_during_retention_process
• code_separation_in_modules
• commons_classes_move
• copy-job-refactor
• datastore_export_doc
• execute_query_method_fix
• export_ds_to_bq
• filtering_restoration_project_in_sli
• handling_and_retrying_google_503_errors
• logging_reduction
• master
• method_reordering_refactor
• move_big_query_package
• new_slo_module
• numBytesRepair
• on_demand_table_backup
• partition_tasks_scheduling_optmization
• problem_with_filtering_one_table_should_not_spoil_the_whole_SLI_processing
• refactored_copy_job_service_tests
• removing_sync_copy_job_scheduler_service
• repair_lastModified_issue
• repair_sli_views_after_schema_migration
• restoration_daily_test_fix
• retention_process_optimization
• services_package_refactoring
• setup-update
• setup_update
• setup_update_Y903
• simplify_setup
• sli_latency_fix
• sli_latency_view_refactor
• support_for_asia_northeast1_region
• tablesWithoutSchemaShouldNotBeBackedUp
• test
• travis_test
• ui_remove_env
• unit_test_packages_refactoring
• views_labels
• y909_ui_improvements
• yacht_972_improved_logging

Build # 239

Build Type

Pull #13

travis-ci

Committed by web-flow

Commit Message

Increasing retries and delay to make 500 errors causes by Datastore eventual consistency less likely to happen.

Details:
When table entity doesn’t exist, then we create proper table entity, and based on that entity we schedule proper copy job.

Based on that copy job couldn’t be scheduled before creation of table entity, after copy job assumes that table entity exist.

Not having table entity is treaded as an error → if we really don’t have table entity for backup what we made, then it means that we have inconsistent data in Datastore.

In existing cases, from eventual consistency of Datastore we don’t get existing table_entity and we retried request till the success.

Existing errors shown that this request was retried 5 times before task fails.
Fortunately, whole task was retried and 3rd attempt of retry was successful.

As error is not repeatitable (occurs randomly in time, probably being dependent from datastore servers load), I increased number of retries and make it in longest period of time to make this kind of errors less likely to happen.

Pull Request Pull Request #13: Increasing retries and delay to make 500 errors causes by Datastore e…

Run Details

1 of 1 new or added line in 1 file covered. (100.0%)

1996 of 2273 relevant lines covered (87.81%)

0.88 hits per line