Logstash-client failed uploading to S3

[JohnNguyen141]

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

• Version: Logstash client 7.0.1
• Operating System: Ubuntu 16
• Config File (if you have sensitive info, please remove it):
  input {
  stdin { }
  }

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
stdout { codec => rubydebug }
s3 {
aws_credentials_file => "/opt/logstash/lNAME_OF_CREDENTIAL.yml"
region => "us-west-2"
bucket => "st-merchant-logos"
prefix => "syslog/%{logstash_host}/"
time_file => 1
tags => ["syslog"]
validate_credentials_on_root_bucket => false
}
}

• Sample Data:
  run logstash-client

/usr/share/logstash/bin/logstash -f logstash.conf

• Steps to Reproduce:
  a) Created a simple logstash conf file as above
  b) > /usr/share/logstash/bin/logstash -f logstash.conf
  c) Paste the input
  127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"

Error:
[INFO ] 2020-09-21 21:44:14.171 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"@timestamp" => 2013-12-11T08:01:45.000Z,
"agent" => ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"",
"message" => "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php\" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"",
"referrer" => ""http://cadenza/xampp/navi.php\"",
"bytes" => "3891",
"host" => "ip-10-1-26-91",
"clientip" => "127.0.0.1",
"ident" => "-",
"verb" => "GET",
"@Version" => "1",
"response" => "200",
"auth" => "-",
"timestamp" => "11/Dec/2013:00:01:45 -0800",
"request" => "/xampp/status.php",
"httpversion" => "1.1"
}
[ERROR] 2020-09-21 21:46:31.582 [S3 output uploader, file: /tmp/logstash/c7d09e78-cb31-4790-a30c-a3e5d581196c/syslog/%{logstash_host}/ls.s3.69ef548f-7423-4074-9fd1-951565c84e98.2020-09-21T21.44.tag_syslog.part0.txt] s3 - Uploading failed, retrying. {:exception=>Seahorse::Client::NetworkingError, :message=>"Net::OpenTimeout", :path=>"/tmp/logstash/c7d09e78-cb31-4790-a30c-a3e5d581196c/syslog/%{logstash_host}/ls.s3.69ef548f-7423-4074-9fd1-951565c84e98.2020-09-21T21.44.tag_syslog.part0.txt", :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:41:in ssl_socket_connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:985:in connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:919:in start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83:in method_missing'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/net_http/connection_pool.rb:285:in start_session'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/net_http/connection_pool.rb:92:in session_for'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/net_http/handler.rb:119:in session'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/net_http/handler.rb:71:in transmit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/net_http/handler.rb:45:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/plugins/content_length.rb:12:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_request_signer.rb:88:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_request_signer.rb:23:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_host_id.rb:14:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/xml/error_handler.rb:8:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:108:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:139:in retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:122:in retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:110:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:139:in retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:122:in retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:110:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:139:in retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:122:in retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/retry_errors.rb:110:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_md5s.rb:31:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/rest/handler.rb:7:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/user_agent.rb:12:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/endpoint_pattern.rb:27:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/endpoint_discovery.rb:67:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/plugins/endpoint.rb:41:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/param_validator.rb:21:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/plugins/raise_response_errors.rb:14:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/aws-sdk-core/plugins/param_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/plugins/response_target.rb:21:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/request.rb:70:in send_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.263/lib/seahorse/client/base.rb:207:in block in define_operation_methods'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.263/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in block in put_object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.263/lib/aws-sdk-resources/services/s3/file_uploader.rb:49:in open_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.263/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:in put_object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.263/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in upload'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.263/lib/aws-sdk-resources/services/s3/object.rb:252:in upload_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.9/lib/logstash/outputs/s3/uploader.rb:38:in upload'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.9/lib/logstash/outputs/s3/uploader.rb:29:in block in upload_async'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.5/lib/concurrent/executor/java_executor_service.rb:85:in `run'"]}

Note: I could manually upload files to S3 since the S3 has public access
Example:
root@IP M:/tmp/logstash/cdf5bee0-9dc4-43b7-9212-25ba8e622fd6/syslog/%{logstash_host}# ls
ls.s3.204b37b5-9a49-4133-8a56-dc1c7683f297.2020-09-21T21.45.tag_syslog.part1.txt

aws s3 cp ls.s3.204b37b5-9a49-4133-8a56-dc1c7683f297.2020-09-21T21.45.tag_syslog.part1.txt s3://st-merchant-logos
upload: ./ls.s3.204b37b5-9a49-4133-8a56-dc1c7683f297.2020-09-21T21.45.tag_syslog.part1.txt to s3://st-merchant-logos/ls.s3.204b37b5-9a49-4133-8a56-dc1c7683f297.2020-09-21T21.45.tag_syslog.part1.txt

I have been researching why the logstash-client failed uploading to S3. There were posts #143 and few others
but they didn't seem to resolve the issue.

Your help would be appreciated.

[TrogloGeek]

Are you maybe using a proxy when using aws command but not configured it in logstash?