Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

log_file_prefix and date_pattern don't support the use of forward slashes #47

Open
JoshMock opened this issue Aug 20, 2021 · 1 comment
Labels

Comments

@JoshMock
Copy link

Logstash information:

Please include the following information:

  1. Logstash version: 7.13.2
  2. Logstash installation source: Docker
  3. How is Logstash being run: Kubernetes
  4. How was the Logstash Plugin installed: Dockerfile line: RUN ./bin/logstash-plugin install logstash-output-google_cloud_storage

OS version Linux kernel 4.14.174 (I believe the Docker image is based on CentOS)

Description of the problem including expected versus actual behavior:

Putting a / character in log_file_prefix (e.g. log_file_prefix => "my-prefix/") causes i/o to the temp_directory location to fail with errors like the following:

[2021-08-20T10:00:00.000][ERROR][logstash.javapipeline    ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Errno::ENOENT: No such file or directory - /tmp/logstash-gcs/my-prefix/logstash_gcs_cw054563.local_2021-08-20T10:00.part001.log.gz>, :backtrace=>["org/jruby/RubyIO.java:1239:in `sysopen'"

Presumably this is because it's trying to write a file into a subdirectory of temp_directory that doesn't exist yet, assuming that all temp files should be written to one flat directory, and uploaded to GCS as such as well. Similar symptoms arise when putting forward slashes in date_pattern.

See #45 and #46 for further examples of this.

Steps to reproduce:

A pipeline to reproduce:

input { generator { } }
output {
  google_cloud_storage {
    bucket => "my-bucket"
    json_key_file => "/secrets/credentials.json"
    log_file_prefix => "my-prefix/"
  }
}
@pspot2
Copy link

pspot2 commented Oct 31, 2023

Analogous parameter implementation in S3 plugin for reference: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html#plugins-outputs-s3-prefix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants