fluent / fluent-plugin-opensearch / 41759601728

93%

push

github

out_opensearch_data_stream: honor id_key (#200)

* out_opensearch_data_stream: honor id_key

The data stream output wrote every record with an empty create
header, so the id_key value never reached the _id of the document
and duplicate events could not be deduplicated. Set _id from id_key
before remove_keys runs, the same way the regular output does.

Fixes #124

Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>

* out_opensearch_data_stream: validate id_key values and treat 409 as duplicate

Send records whose id_key value is an object, an array or longer than 512
bytes to @ERROR instead of the bulk request, since OpenSearch fails the
whole request for them. Treat an empty string like a missing key.

Emit errors raised while building the bulk request when
emit_error_label_event is on. The block passed to the config reader was
never called, so these records were dropped silently.

Count 409 responses as duplicates, as the regular output does, so a
retried chunk does not report already indexed records as failures.

Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>

---------

Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>

22 of 22 new or added lines in 1 file covered. (100.0%)

1264 of 1356 relevant lines covered (93.22%)

334.19 hits per line