Logflare / logflare / 816536411ece77aab96dcc70fbc6400e810bc523-PR-3643

82%

main: 82%

Build Type

Pull #3643

github

Committed by djwhitt

Commit Message

fix(bigquery): handle nested/heterogeneous list payloads on ingest

Two BigQuery paths crashed on the same blind spot: they inspected only
the head of a list and assumed the rest matched.

EventUtils.wrap_fields/1 (insert path) had clauses only for maps and
{k, v} tuples, so a bare list or scalar element — produced by a
list-of-lists value or a list whose head is a map but whose tail holds
lists/scalars — raised FunctionClauseError and crashed the Broadway
batch-processor stage. Add fallback clauses: bare lists recurse over
their elements, any other value passes through unchanged.

SchemaBuilder (schema-patch path) turned a nested array into a
%TableFieldSchema{} with a tuple type (e.g. {"ARRAY", "STRING"}), which
BigQuery cannot represent and Poison cannot encode, crashing the
per-source Schema updater on patch. Drop just the offending field (tuple
inner type) and filter dropped fields at every collection site, so
sibling fields at any depth still build. to_schema_type/1 is unchanged.

Add regression tests for both paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pull Request Pull Request #3643: fix(bigquery): handle nested/heterogeneous list payloads on ingest

Coverage Stats

10 of 10 new or added lines in 2 files covered. (100.0%)

6 existing lines in 2 files now uncovered.

13438 of 16468 relevant lines covered (81.6%)

5086.0 hits per line