[BUG] UnsupportedCharsetException When Enabling HTTP Compression on ARM in OpenSearch Output Plugin

[Edition-X]

Describe the bug
As soon as http_compression is enabled, the following error appears in the logs:

An unknown error occurred sending a bulk request to OpenSearch (will retry indefinitely) {:message=>"ASCII-8BIT", :exception=>Java::JavaNioCharset::UnsupportedCharsetException

To Reproduce
Steps to reproduce the behaviour:

1. Enable http_compression in Opensearch output plugin configuration.
2. Send any bulk requests.
3. Observe the Logstash logs.

Expected behaviour
No charset-related errors should occur when http_compression is enabled.

Plugins

• logstash-output-opensearch (with AWS IAM auth)

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• AWS OpenSearch: 2.5
• Logstash: tested on 8.14.3-aarch64 and 8.17.0-aarch64
• OS: Amazon Linux 2023 (EC2 instance)

Additional context
logstash.yml

pipeline.workers: 4
pipeline.batch.size: 100
pipeline.ecs_compatibility: disabled
pipeline.ordered: false
dead_letter_queue.enable: true
dead_letter_queue.storage_policy: drop_older
dead_letter_queue.retain.age: 12h
path.dead_letter_queue: /data/logstash_temp
# queue config
queue.type: memory
queue.drain: true
# Handling of non-standard field names
config.field_reference.escape_style: percent

opensearch-output-plugin config

input { pipeline { address => opensearch } }

output {
    opensearch {
        id    => "opensearch-<REDACTED>"
        hosts => ["<REDACTED>:443"]
        auth_type => {
          type => 'aws_iam'
          region => "<REDACTED>"
        }
        ssl => true
        ssl_certificate_verification => false
        manage_template => false
        legacy_template => true
        action => "create"
        index => "%{[@metadata][index_name]}"
        ecs_compatibility => "disabled"
        pipeline => "logstash_pipeline"
        http_compression => true
    }
}

input config

input {
    beats {
      id   => "beats-input"
      port => 5044
      add_field => { "ingestion.input" => "input-otp-beat" }
      type => "log"
  }
  }

output {
  pipeline { send_to => filters_pipeline }
}

filter config

input { pipeline { address => filters_pipeline }  }

filter {
    mutate {
        remove_field => [ "[host]" ]
        remove_field => [ "[_hash]" ]
    }
    # add service.hostname if source is filebeat
    if ![service][hostname] {
        if [agent][name] {
            mutate {
                add_field => {
                    "service.hostname" => "%{[agent][name]}"
                }
            }
        }
    }
    mutate {
        add_field => {
          "ingestion.logstash.name" => "<REDACTED>"
          "[@metadata][index_name]" => "aws-logstash-log-%{+YYYY.MM.dd}"
        }
    }
    if ![appAlias] {
        mutate { add_field => { "appAlias" => "default" } }
    }
    if ![appId] {
        mutate { add_field => { "appId" => "unknown" } }
    }
}

output {
    # Main output
        pipeline { send_to => opensearch }
        # Secondary outputs if any
    }

[Edition-X]

AWS Support confirmed they could replicate the ASCII-8BIT error when using AWS IAM auth (on both x86 and ARM). They’ve acknowledged the issue internally and are looking for a workaround. I’ll update this issue if we learn more.

[oeyh]

@dblock @dlvenable Any thoughts on where to look at for potential issues? Looks like it's only happening when http_compression is enabled and AWS IAM auth is used.

[dblock]

I would start by getting a call stack for the exception, it will show you where the error comes from.

[SageMakerX]

Adding below stack trace of the error -

> [2025-02-04T10:11:12,049][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://search-opensearch-25-mpixic2z7wne3dvvuxvdm7ro4u.us-east-1.es.amazonaws.com:443/"}
[2025-02-04T10:11:12,220][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.5.0) {:version=>2}
[2025-02-04T10:11:12,256][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0xcb54e78 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2025-02-04T10:11:12,262][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:disabled}
[2025-02-04T10:11:12,768][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.51}
[2025-02-04T10:11:12,880][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_aa1fb06370afaf847934c88bdb4d98a6", :path=>["/tmp/logs/*.log"]}
[2025-02-04T10:11:12,886][INFO ][filewatch.observingtail  ][main][05e65087032dca792f2981e8ca370bd41384d238d67031362e16248d7f819a9d] START, creating Discoverer, Watch with file and sincedb collections
[2025-02-04T10:11:12,888][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2025-02-04T10:11:12,900][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2025-02-04T10:11:43,164][ERROR][logstash.outputs.opensearch][main][0b42fd1f4b8d3b6fe1022d919acb8e707062fef0a6a70719993cfe36676aa06a] An unknown error occurred sending a bulk request to OpenSearch (will retry indefinitely) {:message=>"ASCII-8BIT", :exception=>Java::JavaNioCharset::UnsupportedCharsetException, :backtrace=>["java.nio.charset.Charset.forName(java/nio/charset/Charset.java:528)", "org.apache.http.entity.ContentType.create(org/apache/http/entity/ContentType.java:257)", "org.apache.http.entity.StringEntity.<init>(org/apache/http/entity/StringEntity.java:116)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)", "jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(jdk/internal/reflect/NativeConstructorAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)", "java.lang.reflect.Constructor.newInstanceWithCaller(java/lang/reflect/Constructor.java:499)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:480)", "org.jruby.javasupport.JavaConstructor.newInstanceDirect(org/jruby/javasupport/JavaConstructor.java:181)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:935)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "RUBY.sign_aws_request(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:212)", "RUBY.perform_request(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:168)", "RUBY.perform_request_to_url(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/pool.rb:272)", "RUBY.perform_request(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/pool.rb:259)", "RUBY.with_connection(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/pool.rb:348)", "RUBY.perform_request(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/pool.rb:258)", "RUBY.Pool(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/pool.rb:266)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:375)", "RUBY.bulk_send(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client.rb:179)", "RUBY.bulk(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client.rb:157)", "RUBY.safe_bulk(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/plugin_mixins/opensearch/common.rb:226)", "RUBY.submit(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/plugin_mixins/opensearch/common.rb:158)", "RUBY.retrying_submit(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/plugin_mixins/opensearch/common.rb:107)", "RUBY.multi_receive(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch.rb:277)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.invokeOutput(org/logstash/config/ir/compiler/OutputStrategyExt.java:153)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.doOutput(org/logstash/config/ir/compiler/OutputStrategyExt.java:279)", "org.logstash.config.ir.compiler.OutputStrategyExt$SharedOutputStrategyExt.output(org/logstash/config/ir/compiler/OutputStrategyExt.java:312)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.multiReceive(org/logstash/config/ir/compiler/OutputStrategyExt.java:143)", "org.logstash.config.ir.compiler.OutputDelegatorExt.doOutput(org/logstash/config/ir/compiler/OutputDelegatorExt.java:111)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.lambda$multiReceive$0(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "co.elastic.logstash.api.TimerMetric.lambda$time$0(co/elastic/logstash/api/TimerMetric.java:35)", "org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(org/logstash/instrument/metrics/timer/ConcurrentLiveTimerMetric.java:47)", "co.elastic.logstash.api.TimerMetric.time(co/elastic/logstash/api/TimerMetric.java:34)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "org.logstash.generated.CompiledDataset1.compute(org/logstash/generated/CompiledDataset1)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:351)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:341)", "org.logstash.execution.ObservedExecution.lambda$compute$0(org/logstash/execution/ObservedExecution.java:17)", "org.logstash.execution.WorkerObserver.lambda$observeExecutionComputation$0(org/logstash/execution/WorkerObserver.java:39)", "org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(org/logstash/instrument/metrics/timer/ConcurrentLiveTimerMetric.java:47)", "org.logstash.execution.WorkerObserver.lambda$executeWithTimers$1(org/logstash/execution/WorkerObserver.java:50)", "org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(org/logstash/instrument/metrics/timer/ConcurrentLiveTimerMetric.java:47)", "org.logstash.execution.WorkerObserver.executeWithTimers(org/logstash/execution/WorkerObserver.java:50)", "org.logstash.execution.WorkerObserver.observeExecutionComputation(org/logstash/execution/WorkerObserver.java:38)", "org.logstash.execution.ObservedExecution.compute(org/logstash/execution/ObservedExecution.java:17)", "org.logstash.execution.WorkerLoop.abortableCompute(org/logstash/execution/WorkerLoop.java:113)", "org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:86)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:300)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:164)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:354)", "java.lang.Thread.run(java/lang/Thread.java:840)"]}

logstash-output-opensearch configurations used -

> input {
        file {
                path => "/tmp/logs/*.log"
                start_position => beginning
        }
}
output {
        opensearch {
                hosts => "https://search-opensearch-25-mpixic2z7wne3dvvuxvdm7ro4u.us-east-1.es.amazonaws.com:443"
                auth_type => {
                type => 'aws_iam'
                region => 'us-east-1'
                }
                index => "log-%{+YYYY.MM.dd}"
                ecs_compatibility => disabled
                ssl_certificate_verification => false
                http_compression => true
        }
}

[dblock]

Looks like Sigv4 signing doesn't know how to handle compressed content. This is in the call stack:

"RUBY.sign_aws_request(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.3-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:212)"

Needs an implementation.