Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logstash doesn't release logs properly for log rotation #63

Open
reallistic opened this issue Aug 11, 2016 · 1 comment
Open

Logstash doesn't release logs properly for log rotation #63

reallistic opened this issue Aug 11, 2016 · 1 comment

Comments

@reallistic
Copy link

It seems that when log rotation runs and truncates (because copytrunate is set) the file is still open pointing at the end thus causing the file pointer null bytes problem.

This causes the filesystem to think that the truncated file is still the same size, thus rotation happens every time even though the true file size (ls -lsh) is small.
Other programs (like nginx) fix this by sending a signal to the process so it will release the file.
I noticed that logstash is started with nice and thats where stdout and stderr are redirected to the log locations. There should be a way to send a signal to nice or the process it starts to enable this. If I find it, i'll submit a PR.

@spujadas
Copy link
Owner

Some early thoughts.

Logstash's PID is located in /var/run/logstash.pid, so kill -HUP $(cat /var/run/logstash.pid) (e.g. in a postrotate section of the logrotate configuration file) reloads Logstash (used for instance to force it to reload its configuration file), but I don't know if this is enough for it to release the log file.

Alternatively, stopping Logstash (in a prerotate section of the logrotate configuration file) then starting it (in postrotate) should make it release the log file (and should also allow us to get rid of copyrotate, which always seems a bit hackish) but this reduces the availability of the service (as Logstash takes quite some time to start) so doesn't seem fitting for a production system.

Anyway, little to no testing done at this point, so just throwing some ideas around for now.

P.S.: the logrotate configuration in the image is the one from the official Logstash RPM/DEB package, so they probably have the exact same problem (although I haven't found conclusive evidence that this is the case or that it has been addressed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants