Logflare / logflare / 5517924684db4927ee414caea43ca8e1a5deba85

82%

Build Type

push

github

Committed by web-flow

Commit Message

fix(bigquery): handle nested/heterogeneous list payloads on ingest (#3643)

Two BigQuery paths crashed on the same blind spot: they inspected only
the head of a list and assumed the rest matched.

EventUtils.wrap_fields/1 (insert path) had clauses only for maps and
{k, v} tuples, so a bare list or scalar element — produced by a
list-of-lists value or a list whose head is a map but whose tail holds
lists/scalars — raised FunctionClauseError and crashed the Broadway
batch-processor stage. Add fallback clauses: bare lists recurse over
their elements, any other value passes through unchanged.

SchemaBuilder (schema-patch path) turned a nested array into a
%TableFieldSchema{} with a tuple type (e.g. {"ARRAY", "STRING"}), which
BigQuery cannot represent and Poison cannot encode, crashing the
per-source Schema updater on patch. Drop just the offending field (tuple
inner type) and filter dropped fields at every collection site, so
sibling fields at any depth still build. to_schema_type/1 is unchanged.

Add regression tests for both paths.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Coverage Stats

10 of 10 new or added lines in 2 files covered. (100.0%)

2 existing lines in 2 files now uncovered.

13478 of 16509 relevant lines covered (81.64%)

5192.81 hits per line