# Optimizing Celery Worker Performance

Tuning Celery for high throughput depends heavily on whether your tasks are **CPU-bound** (heavy calculation, image processing) or **I/O-bound** (network requests, database queries).

## 1. Concurrency Pools

The default execution pool is `prefork`, which forks a separate process for each concurrent task. This is robust but heavy on memory.

### CPU-Bound Tasks
Stick to the default **prefork** pool. The concurrency level should roughly match your CPU core count.

```bash
# Default behavior (usually number of CPUs)
celery -A tasks worker --pool=prefork --concurrency=4
```

### I/O-Bound Tasks
If your tasks spend most of their time waiting for HTTP requests or DB queries, use **gevent** or **eventlet**. These use greenlets (lightweight threads) allowing a single process to handle hundreds or thousands of concurrent tasks.

```bash
pip install gevent
celery -A tasks worker --pool=gevent --concurrency=1000
```

## 2. Prefetch Multiplier

The `worker_prefetch_multiplier` setting determines how many messages a worker reserves from the broker at once.

*   **Default (4):** Good for tasks with mixed durations.
*   **Long-running tasks:** Set to **1**. This prevents one worker from hogging tasks while others sit idle.
*   **Short, fast tasks:** Increase to **10** or more. This reduces the overhead of round-trips to the broker.

```python
# In celery config
app.conf.worker_prefetch_multiplier = 10
```

## 3. Ignoring Results

Storing task results in a backend (Redis/DB) adds significant overhead (network I/O + serialization). If you don't need the return value, disable it.

### Global Setting
```python
app.conf.task_ignore_result = True
```

### Per-Task Setting
```python
@app.task(ignore_result=True)
def fire_and_forget_task():
    # ...
    pass
```

## 4. Serialization Protocol

JSON is the default and is generally fast, but for extremely high throughput, consider **MessagePack** or **Pickle** (if security allows).

*   **Pickle:** Fastest for Python objects, but insecure (don't use if you accept tasks from untrusted sources).
*   **Msgpack:** Binary serialization, faster and smaller than JSON.

```bash
pip install msgpack
```

```python
app.conf.task_serializer = 'msgpack'
app.conf.result_serializer = 'msgpack'
app.conf.accept_content = ['msgpack']
```

## 5. Connection Pooling

If you are using a database backend (like PostgreSQL) or Redis, ensure connection pooling is enabled and sized correctly so workers don't spend time establishing new connections for every task.

For Redis, Celery handles this well by default, but ensure your Redis server `maxclients` setting is high enough to handle `concurrency * number_of_workers`.

[[programming/python/celery]]