Logflare / logflare / f677e81f9625f9d4490323efe7d081aab525cb8a

80%

Build Type

push

github

Committed by web-flow

Commit Message

feat: add drop_fields ingest transform (#3449)

* bench: cover copy + kv configurations in log_event_make_bench

Adds copy_fields and kv_enrich scenarios for both the otel trace and edge
log payloads, plus a "copy + kv" full-pipeline scenario, so realistic-shape
per-transform cost shows up alongside the existing baseline scenarios. KV
inserts cover the "project" key value in each payload so the kv_enrich
rules actually fire.

Designed to be portable across history: configures copy_fields via the
raw config string only (no parse_copy_fields_config call), so the bench
runs cleanly on commits before any of this branch's perf or feature
changes. Drop fields coverage is added separately later, after the
drop_fields feature itself lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: add drop fields ingest transform

Add a third source-level transform alongside copy fields and key-value
enrichment, letting users remove specific keys from event bodies at
ingestion. The motivating case is keys used only upstream for routing
(e.g. Fluent Bit's `service` / `namespace`) that should not be persisted.

Drop fields runs last in `LogEvent.transform/2` so values can be copied
or used for enrichment before being dropped from their original location.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf: pre-parse drop_fields config to avoid per-event reparsing

Mirrors the existing transform_key_values_parsed pattern: the raw
textarea config is parsed once when a Source is loaded into the cache
and stored on the new transform_drop_fields_parsed virtual field, so
the ingest hot path walks a list of path segment lists instead of
re-splitting and re-trimming the config string for every event.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf: pre-parse copy_fields config to avoid per-event reparsing

Applies the same parsed-virtual-field pattern used for key_values and
drop_fields: the ra... (continued)

Coverage Stats

49 of 51 new or added lines in 3 files covered. (96.08%)

2 existing lines in 1 file now uncovered.

12420 of 15592 relevant lines covered (79.66%)

15839.22 hits per line