Logflare / logflare / 62aaa6ef0980ea9f7ff66d2dc8c147243edafc68

81%

Build Type

push

github

Committed by web-flow

Commit Message

perf: rewrite BigQuerySchemaChange.validate as single-pass walk (#3489)

* perf: rewrite BigQuerySchemaChange.validate as single-pass walk

The previous implementation built a full metadata flat typemap from
the body via SchemaUtils.to_typemap/1 + SchemaUtils.flatten_typemap/1,
then Map.merged it against the cached schema flat map purely to detect
type conflicts. The flat map was discarded immediately after.

Walk the body once and check each leaf's inferred type against the
schema flat map key-by-key. Build dot-delimited paths inline, raise
on type mismatch, return :ok if no conflict. No intermediate typemap,
no flatten pass, no allocation of the throwaway map.

Behavior changes / fixes exposed by the rewrite:

  * validate/2 was previously passing the raw %SourceSchema{} struct
    (not its schema_flat_map field) into the merge, so atom struct
    keys never overlapped with string flat-map keys and type
    conflicts were silently ignored. Now we extract :schema_flat_map
    and actually compare. Mismatches that production used to swallow
    will now surface as {:error, _} at validate time (previously they
    failed downstream at BigQuery insert time anyway).
  * "timestamp" is skipped at the top level because LogEvent.mapper/1
    injects it as integer microseconds, while the BQ schema types it
    as TIMESTAMP (:datetime). Without the skip, every event with a
    cached schema would now fail validation.
  * Short-circuit when schema_flat_map is empty (cold-start sources)
    so we don't walk the body just to find nothing.
  * try_merge / merge_flat_typemaps removed — they were internal
    helpers tied to the old shape.

Test file un-failed: the @moduletag :failing was hiding pre-existing
correctness bugs (valid? was called with metadata-only maps against
full schemas, so no key ever matched). Tests rewritten to wrap with
%{"metadata" => ...} so they actually exercise the validator paths
they claim to test; the rewrite makes them all pass.

Co-A... (continued)

Coverage Stats

45 of 47 new or added lines in 5 files covered. (95.74%)

2 existing lines in 2 files now uncovered.

12629 of 15706 relevant lines covered (80.41%)

3761.44 hits per line