Logflare / logflare / de8a820cc27a4efe435cfc280b6f393966b36897

81%

Build Type

push

github

Committed by web-flow

Commit Message

fix+perf: removes DateTime usage from BQ hot code path, unix ns handling (#3557)

* fix+perf: removes DateTime usage from BQ hot code path, and refines the schema determination for otel timestamp fields

* chore: PR comments

* fix: share one OTel timestamp predicate between BQ schema typing and value conversion (#3742)

* fix: share one OTel timestamp predicate between BQ schema typing and value conversion

The BigQuery path had two independent definitions of "start_time/end_time
is a timestamp": SchemaBuilder typed the column via TypeDetection.detect/1
while convert_to_seconds/1 converted values on field name + magnitude alone.
When they disagreed (e.g. a plain log with a nanosecond-range integer
start_time), a float value was ingested into an INTEGER-typed column.

Introduce TypeDetection.otel_timestamps?/1 as the single shared predicate
(detected :metric/:trace, or OTel-shaped resource+scope payloads) and use it
in both paths. le_to_bq_row/1 and log_event_to_df_struct/1 compute it on the
raw event body before ingest transformation, since map-wrapping breaks the
metadata.type fast path.

Also fixes the convert_to_seconds/2 microsecond clause writing the timestamp
value into the field under conversion, and restores the resource+scope
behavior asserted by schema_builder_test (failing on CI).


* fix: detect otel_timestamps once at ingest and store on LogEvent

Avoids re-running OTel detection in the BQ hot path: LogEvent.make/3 and
make_from_spool/2 compute the flag once (reusing the already-detected
event_type) and le_to_bq_row/1 / log_event_to_df_struct/1 read it off the
struct. Also use :math.pow for the second-conversion divisors.


* fix: narrow otel_timestamps to the metric/trace signal and condense conversion

Drops the resource+scope shape heuristic - otel_timestamps?/1 now accepts raw
params or a precomputed event type and only honors detected :metric/:trace.
OTel test payloads carry metadata.type, matching what the OTEL processors set.
Cond... (continued)

Coverage Stats

13 of 15 new or added lines in 4 files covered. (86.67%)

7 existing lines in 3 files now uncovered.

13631 of 16731 relevant lines covered (81.47%)

11948.6 hits per line