You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using tcmalloc will help us to avoid memory leaks when memory fragmentation existed
TL;DR;
The glibc malloc is the most popular memory allocator in most distributions of Linux. We use malloc to allocate memory from the system and free to return the memory to the system. But there is a problem. When we call free , the memory may not return to system immediately. The glibc malloc will cache the small memory into a buffer and return it into system when it become a big chunk. But if the memory we allocate from the system is too small and too split to merge into a big chunk, the memory will be in used status for a long time even if we have already return it. We call it memory leak.
But we always use a lot of C extensions for Python. The extension may not use the Python memory allocator and use malloc and free API directly. So the memory leak again.
Feature request
Related to #4760
Using tcmalloc will help us to avoid memory leaks when memory fragmentation existed
TL;DR;
The glibc malloc is the most popular memory allocator in most distributions of Linux. We use
malloc
to allocate memory from the system andfree
to return the memory to the system. But there is a problem. When we callfree
, the memory may not return to system immediately. The glibc malloc will cache the small memory into a buffer and return it into system when it become a big chunk. But if the memory we allocate from the system is too small and too split to merge into a big chunk, the memory will be in used status for a long time even if we have already return it. We call it memory leak.In Python, the developer warp the glibc allocator use the
munmap
API to ensure the memory returned into system immediately. FYI https://github.com/python/cpython/blob/main/Objects/obmalloc.c#L380-L395. So the memory leak is fine when our project is pure Python codeBut we always use a lot of C extensions for Python. The extension may not use the Python memory allocator and use
malloc
andfree
API directly. So the memory leak again.FYI protocolbuffers/protobuf#10088 (comment)
So we have two way to solve this
malloc_trim
background periodicallytcmalloc
instead of the glibc mallocI perfer 2
Motivation
No response
Other
No response
The text was updated successfully, but these errors were encountered: