fluent / fluent-plugin-opensearch / 26486667552

91%

push

github

fix: filter failed items in data stream bulk error log (#175)

* fix: filter failed items in data stream bulk error log

When response['errors'] is true, log only the items with status >= 300
instead of the entire bulk response. This prevents log lines with
thousands of characters when a small number of documents fail in a
large batch.

Signed-off-by: Rodrigo Carvalho <rodrigo.carvalho@bemobi.com>

* test: add tests for bulk error log filtering in data stream

Verify that when response['errors'] is true, only items with
status >= 300 appear in the error log, not successful items.

Signed-off-by: Rodrigo Carvalho <rodrigo.carvalho@bemobi.com>

* fix: emit failed records to @ERROR label on bulk insert failure

Replace logging of full failed items array with router.emit_error_event
for each failed record, allowing users to handle errors via <label @ERROR>.
A summary log with the failure count is kept for observability.
Respects the emit_error_label_event config flag (default: true).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Rodrigo Carvalho <rodrigo.carvalho@bemobi.com>

* test: use driver.logs for capturing plugin log output

The new bulk error tests were checking Fluent::Engine.log.out.logs,
but plugin log.error calls go to the test driver's logger. Switch to
driver.logs (the pattern used elsewhere in this test suite) so the
"Could not bulk insert" assertions can find the log entry.

Signed-off-by: Rodrigo Carvalho <rodrigo.carvalho@bemobi.com>

---------

Signed-off-by: Rodrigo Carvalho <rodrigo.carvalho@bemobi.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

12 of 12 new or added lines in 1 file covered. (100.0%)

1173 of 1282 relevant lines covered (91.5%)

312.1 hits per line