Logflare / logflare / 7b2d6c250992566b79282cd19e1012144637eb69

81%

push

github

fix: stop returning 500 from OTLP ingest when validation rejects events (#3532)

* fix: stop returning 500 from OTLP ingest when validation rejects events

OTEL HTTP controllers (`otel_metrics`, `otel_traces`, `otel_logs`) piped
`Processor.ingest`'s `{:ok, count} | {:error, errors}` return value
through `protobuf_response/3`, which mapped the error case to 500.

`Backends.ingest_logs/3` returns `{:error, errors}` whenever *any* event
in the batch fails validation (`split_valid_events` accumulates each
invalid event's pipeline-error message). So a single bad event in a
batch returned 500 for the whole POST, which OTEL exporters then retry
under their backoff policy.

This was dormant until the BigQuerySchemaChange validator started
actually surfacing type conflicts. Previously the validator silently
swallowed mismatches (it merged a `%SourceSchema{}` struct against the
flat-map string keys, so nothing ever overlapped), and `errors` was
effectively always empty. Once the validator started returning
`{:error, _}` correctly, the controller's pre-existing strict mapping
turned into customer-visible 5xx storms — especially on metrics, where
the OTLP `as_int`/`as_double` oneof drifts numeric fields between
integer and float on the wire.

Changes:

- `protobuf_response({:error, _}, ...)` now logs a sampled warning and
  acks with 200. Matches the non-OTEL `handle/2` path's intent (ack
  invalid events into the rejected stream rather than fail the batch)
  and stops the exporter retry storm. The rescue clause for genuine
  uncaught exceptions still 500s — only validation rejections are
  treated as non-fatal.
- gRPC OTEL servers (`LogflareGrpc.{Metrics,Trace,Logs}.Server`) gain
  the same warn-and-ack pattern. They previously ignored the return
  value entirely, so silent drops had zero observability.
- Relax `BigQuerySchemaChange.enforce_type/3` so `:integer` satisfies
  a `:float` schema slot (and `{:list, :integer}` satisfies
  `{:list, :float}`). BQ implic... (continued)

20 of 29 new or added lines in 5 files covered. (68.97%)

12711 of 15790 relevant lines covered (80.5%)

3802.47 hits per line