fluent / fluent-plugin-opensearch

93%

LAST BUILD ON BRANCH main

branch: main

CHANGE BRANCH

x

Reset

• main
• add-constraint-for-faraday-middleware
• add-ism-documentation-in-troubleshooting-doc
• add-testcases-for-hosts-parameter
• add-to-project
• adjust-github-workflows
• align-lowercases
• allow_duplicate_key
• ci-add-ruby-3.2
• ci-grouping-pr
• ci-ruby-3.2-minimum
• ci-sha
• configurable-unrecoverable-record-types
• dependabot.yml
• dependabot/github_actions/actions/add-to-project-2.0.0
• dependabot/github_actions/actions/checkout-6.0.3
• dependabot/github_actions/actions/create-github-app-token-3.1.1
• dependabot/github_actions/actions/create-github-app-token-3.2.0
• dependabot/github_actions/coverallsapp/github-action-2.3.6
• dependabot/github_actions/monthly-updates-27f38f1e4e
• dependabot/github_actions/monthly-updates-a7c1315158
• dependabot/github_actions/monthly-updates-b381ee1f94
• dependabot/github_actions/monthly-updates-e631a66382
• dependabot/github_actions/roots/issue-closer-action-1.2
• dependabot/github_actions/ruby/setup-ruby-1.299.0
• dependabot/github_actions/ruby/setup-ruby-1.306.0
• dependabot/github_actions/ruby/setup-ruby-1.310.0
• docs
• fix-badge
• fix-customize-template
• fix-data-stream-id-key
• fix-flush
• fix-host
• fix-memory-usage
• fix-secret
• fix/data-stream-bulk-error-logging
• fix_duplicate_keys_in_test_fixtures
• fix_expired_sigv4_token
• handle-exceptions-on-credentials-retriving
• handle-opensearch-serverless
• handle-suppress_type_name-correctly
• implement-aws-opensearch-sign-request-mechanism
• issue-template
• load-faraday-v2
• obtain-aws-credentials-on-every-requests-if-available
• pass-credentials-duration
• permit-to-handle-hosts-parameter
• prevent_refresh_from_stopping_on_exception
• refresh_credentials_interval
• refs/tags/v1.0.0
• refs/tags/v1.0.1
• refs/tags/v1.0.10
• refs/tags/v1.0.2
• refs/tags/v1.0.3
• refs/tags/v1.0.4
• refs/tags/v1.0.5
• refs/tags/v1.0.6
• refs/tags/v1.0.7
• refs/tags/v1.0.8
• refs/tags/v1.0.9
• refs/tags/v1.1.0
• refs/tags/v1.1.1
• refs/tags/v1.1.2
• refs/tags/v1.1.3
• refs/tags/v1.1.4
• refs/tags/v1.1.5
• release-1.2.0
• release-v1.0.0
• release-v1.1.8
• revert-duration
• ruby_versions.yml
• specify-host-argument-if-needed
• suppress-emit-error-label-events
• suppress_type_name
• tmp
• update-ci
• upload-coverage-into-coveralls
• use-faraday-1
• use-faraday-2
• use-if-clause-to-detect-requirements-for-emitting-error-label-events
• v1.1.6
• v1.1.7
• v1.1.8
• webrick
• yajl-to-json

Build # 41759601728

Build Type

push

github

Committed by web-flow

Commit Message

out_opensearch_data_stream: honor id_key (#200)

* out_opensearch_data_stream: honor id_key

The data stream output wrote every record with an empty create
header, so the id_key value never reached the _id of the document
and duplicate events could not be deduplicated. Set _id from id_key
before remove_keys runs, the same way the regular output does.

Fixes #124

Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>

* out_opensearch_data_stream: validate id_key values and treat 409 as duplicate

Send records whose id_key value is an object, an array or longer than 512
bytes to @ERROR instead of the bulk request, since OpenSearch fails the
whole request for them. Treat an empty string like a missing key.

Emit errors raised while building the bulk request when
emit_error_label_event is on. The block passed to the config reader was
never called, so these records were dropped silently.

Count 409 responses as duplicates, as the regular output does, so a
retried chunk does not report already indexed records as failures.

Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>

---------

Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>

Coverage Stats

22 of 22 new or added lines in 1 file covered. (100.0%)

1264 of 1356 relevant lines covered (93.22%)

334.19 hits per line