Python Memory Management
Memory management in Python involves a private heap containing all Python objects and data structures. The management of this private heap is ensured internally by the Python memory manager.
The Private Heap
The private heap is where all Python objects and data structures are stored. The programmer does not have access to this heap directly; instead, the Python memory manager handles it.
Python Memory Manager
The Python memory manager has different components which deal with various dynamic storage management aspects, like sharing, segmentation, preallocation or caching.
Hierarchy
- Raw Memory Allocator: Interacts with the memory manager of the operating system.
- Object-Specific Allocators: Operates on the same heap and implements distinct memory management policies adapted to the peculiarities of every object type.
Reference Counting
Python uses reference counting as its primary memory management mechanism.
- Every object has a reference count.
- The count increases when an object is referenced (e.g., assigned to a variable).
- The count decreases when a reference is deleted or goes out of scope.
- When the count reaches zero, the memory is deallocated.
import sys
a = []
b = a
# The reference count is higher than expected because getrefcount() creates a temporary reference
print(sys.getrefcount(a)) Garbage Collection
While reference counting handles most cases, it cannot handle reference cycles (e.g., object A references B, and B references A). Python has a cyclic garbage collector to detect and clean up these cycles.
See programming/python/garbage-collection for more details.
Memory Pools (pymalloc)
Python has a specialized allocator for small objects (less than 512 bytes) called pymalloc. It uses memory pools to reduce fragmentation and improve performance.
- Blocks: The smallest unit of allocation.
- Pools: A collection of blocks of the same size class.
- Arenas: A collection of pools (typically 256KB chunks).