googlegenomics / gcp-variant-transforms / 659

89%

Pull #197

travis-ci

Fixing BigQuery "Too many sources provided" error

We observed during multiple runs of VT pipeline where we set the --num_workers to
high number (for example 256), it fails with the following error
message:

    BigQuery execution failed., Error:
    Message: Too many sources provided: xxxxx. Limit is 10000
This solution was proposed here:
https://stackoverflow.com/questions/44255924/error-message-too-many-sources-provided-15285-limit-is-10000

According to beam's developers:
"BQ sink failure to producing too many files (shards) is a known issue ... hoping to add a BQ custom sink to Python SDK which will fix this. Relevant JIRA is https://issues.apache.org/jira/browse/BEAM-2801."

64 of 64 new or added lines in 3 files covered. (100.0%)

4052 of 4439 relevant lines covered (91.28%)

0.91 hits per line