Debugging Memory Leaks in Celery Workers
Memory leaks in Celery workers are a common issue because workers are long-running processes. If a task fails to release memory properly, that memory remains occupied until the worker process restarts. Over time, this can consume all available RAM.
1. The "Quick Fix": Restarting Workers
The easiest way to mitigate memory leaks without finding the root cause is to configure Celery to restart worker processes periodically.
worker_max_tasks_per_child
This setting tells the worker to restart a child process after it has executed a specific number of tasks.
# Restart worker process after 1000 tasks
app.conf.worker_max_tasks_per_child = 1000worker_max_memory_per_child
This setting tells the worker to restart a child process if it exceeds a certain memory limit (in Kilobytes).
# Restart if memory usage exceeds 200MB (200,000 KB)
app.conf.worker_max_memory_per_child = 200000Note: These are band-aid solutions. They prevent the server from crashing but do not fix the underlying leak.
2. Using tracemalloc (Built-in)
Python's built-in tracemalloc module can track memory blocks allocated by Python. You can use it inside a task to see what is consuming memory.
import tracemalloc
from celery import Celery
app = Celery('tasks', broker='redis://localhost')
@app.task
def leaky_task():
tracemalloc.start()
# ... run your logic ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[ Top 10 ]")
for stat in top_stats[:10]:
print(stat)3. Using memray (Linux/macOS)
memray is a high-performance memory profiler for Python. It is excellent for visualizing memory usage over time.
- Install memray:
pip install memray - Run the Celery worker under memray:
memray run -o output.bin -m celery -A tasks worker --loglevel=INFO- After stopping the worker, generate a report:
memray flamegraph output.bin4. Common Causes
- Global Variables: Appending data to a global list or dictionary without ever clearing it.
- Caching Clients: Creating a new database or Redis connection inside a task but never closing it.
- Circular References: Objects that reference each other, preventing the reference count from reaching zero (though Python's GC usually handles this, complex cycles involving
__del__can be problematic).