Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestion to add a potential memory leak warning to the readme #29

Open
moshetet opened this issue May 30, 2019 · 6 comments
Open

Suggestion to add a potential memory leak warning to the readme #29

moshetet opened this issue May 30, 2019 · 6 comments

Comments

@moshetet
Copy link

moshetet commented May 30, 2019

S3StreamLogger will keep the logs in this.buffers until it needs to create a new file.
So if setting large rotate_every and max_file_size you can bump into a nasty memory leak.

A warning should be added to the README to warn against this, especially since the following 2 things cause you to believe it is not a possibility unless you dig into the code of the package:

  1. The package says that it implements a stream (which usually means things don't stay in memory) after they are sent away.
  2. The options rotate_every and max_file_size just refer to the way the files are rotated in S3.
@moshetet moshetet changed the title suggestion to add a potential memory leak warning to the readme Suggestion to add a potential memory leak warning to the readme May 30, 2019
@autopulated
Copy link
Member

In order to update an existing file in S3, this module re-uploads the entire file. Due to the way S3 works, it is not practical to do anything else. For this reason you should use a relatively small maximum file size.

@conor-deegan
Copy link

Once a file rotates or reaches it's max_file_size or rotates in S3, are the logs cleared from this.buffer?

Once a file rotates or reaches it's maximum do the logs on the application clear?

@autopulated
Copy link
Member

Yes, buffers are cleared when a file is rotated due to reaching maximim age or size.

@conor-deegan
Copy link

Thank you!

@vks-dbb
Copy link

vks-dbb commented Jan 10, 2024

is there might be some potential logs losing with parallel executing?

@autopulated
Copy link
Member

@vks-dbb Yes, different servers/processes should write log files to different files. The default name_format tries to ensure this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants