657
89%

Ran 20 Apr 2018 04:18PM UTC

Jobs 1

Files 60

Run time 4s

Badge

Embed ▾

pending completion

Build # 657

Build Type

Pull #197

travis-ci

Committed by

web-flow

Commit Message

Fixing BigQuery "Too many sources provided" error

We observed during multiple runs of VT pipeline where we set the --num_workers to
high number (for example 256), it fails with the following error
message:

    BigQuery execution failed., Error:
    Message: Too many sources provided: xxxxx. Limit is 10000
This solution was proposed here:
https://stackoverflow.com/questions/44255924/error-message-too-many-sources-provided-15285-limit-is-10000

According to beam's developers:
"BQ sink failure to producing too many files (shards) is a known issue ... hoping to add a BQ custom sink to Python SDK which will fix this. Relevant JIRA is https://issues.apache.org/jira/browse/BEAM-2801."

Pull Request Pull Request #197: Fixing BigQuery "Too many sources provided" error

Run Details

27 of 27 new or added lines in 2 files covered. (100.0%)

4005 of 4402 relevant lines covered (90.98%)

0.91 hits per line

New Missed Lines in Diff

Lines	Coverage	∆	File
5	64.1	-5.59%	gcp_variant_transforms/transforms/variant_to_bigquery.py
9	55.0		gcp_variant_transforms/transforms/limit_write.py

Jobs

ID	Job ID	Ran	Files	Coverage
1	657.1	20 Apr 2018 04:18PM UTC	60	90.98	Travis Job 657.1

googlegenomics / gcp-variant-transforms / 657
89%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

New Missed Lines in Diff

Jobs

Source Files on build 657

googlegenomics / gcp-variant-transforms / 657 89%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

New Missed Lines in Diff

Jobs

Source Files on build 657

googlegenomics / gcp-variant-transforms / 657
89%

README BADGES
x