# Debugging Memory Leaks in Celery Workers

Memory leaks in Celery workers are a common issue because workers are long-running processes. If a task fails to release memory properly, that memory remains occupied until the worker process restarts. Over time, this can consume all available RAM.

## 1. The "Quick Fix": Restarting Workers

The easiest way to mitigate memory leaks without finding the root cause is to configure Celery to restart worker processes periodically.

### `worker_max_tasks_per_child`

This setting tells the worker to restart a child process after it has executed a specific number of tasks.

```python
# Restart worker process after 1000 tasks
app.conf.worker_max_tasks_per_child = 1000
```

### `worker_max_memory_per_child`

This setting tells the worker to restart a child process if it exceeds a certain memory limit (in Kilobytes).

```python
# Restart if memory usage exceeds 200MB (200,000 KB)
app.conf.worker_max_memory_per_child = 200000
```

*Note: These are band-aid solutions. They prevent the server from crashing but do not fix the underlying leak.*

## 2. Using `tracemalloc` (Built-in)

Python's built-in `tracemalloc` module can track memory blocks allocated by Python. You can use it inside a task to see what is consuming memory.

```python
import tracemalloc
from celery import Celery

app = Celery('tasks', broker='redis://localhost')

@app.task
def leaky_task():
    tracemalloc.start()
    
    # ... run your logic ...
    
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    
    print("[ Top 10 ]")
    for stat in top_stats[:10]:
        print(stat)
```

## 3. Using `memray` (Linux/macOS)

`memray` is a high-performance memory profiler for Python. It is excellent for visualizing memory usage over time.

1.  Install memray: `pip install memray`
2.  Run the Celery worker under memray:

```bash
memray run -o output.bin -m celery -A tasks worker --loglevel=INFO
```

3.  After stopping the worker, generate a report:

```bash
memray flamegraph output.bin
```

## 4. Common Causes

1.  **Global Variables:** Appending data to a global list or dictionary without ever clearing it.
2.  **Caching Clients:** Creating a new database or Redis connection inside a task but never closing it.
3.  **Circular References:** Objects that reference each other, preventing the reference count from reaching zero (though Python's GC usually handles this, complex cycles involving `__del__` can be problematic).

[[programming/python/celery]]