Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Use tcmalloc in supported platform as default memory allocator #4766

Open
Zheaoli opened this issue May 30, 2024 · 0 comments
Open
Labels
enhancement Enhancement proposals

Comments

@Zheaoli
Copy link

Zheaoli commented May 30, 2024

Feature request

Related to #4760

Using tcmalloc will help us to avoid memory leaks when memory fragmentation existed

TL;DR;

The glibc malloc is the most popular memory allocator in most distributions of Linux. We use malloc to allocate memory from the system and free to return the memory to the system. But there is a problem. When we call free , the memory may not return to system immediately. The glibc malloc will cache the small memory into a buffer and return it into system when it become a big chunk. But if the memory we allocate from the system is too small and too split to merge into a big chunk, the memory will be in used status for a long time even if we have already return it. We call it memory leak.

In Python, the developer warp the glibc allocator use the munmap API to ensure the memory returned into system immediately. FYI https://github.com/python/cpython/blob/main/Objects/obmalloc.c#L380-L395. So the memory leak is fine when our project is pure Python code

But we always use a lot of C extensions for Python. The extension may not use the Python memory allocator and use malloc and free API directly. So the memory leak again.

FYI protocolbuffers/protobuf#10088 (comment)

So we have two way to solve this

  1. Binding glibc and run malloc_trim background periodically
  2. Use tcmalloc instead of the glibc malloc

I perfer 2

Motivation

No response

Other

No response

@Zheaoli Zheaoli added the enhancement Enhancement proposals label May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement proposals
Projects
None yet
Development

No branches or pull requests

1 participant