Optimizing Celery Worker Performance
Tuning Celery for high throughput depends heavily on whether your tasks are CPU-bound (heavy calculation, image processing) or I/O-bound (network requests, database queries).
1. Concurrency Pools
The default execution pool is prefork, which forks a separate process for each concurrent task. This is robust but heavy on memory.
CPU-Bound Tasks
Stick to the default prefork pool. The concurrency level should roughly match your CPU core count.
# Default behavior (usually number of CPUs)
celery -A tasks worker --pool=prefork --concurrency=4I/O-Bound Tasks
If your tasks spend most of their time waiting for HTTP requests or DB queries, use gevent or eventlet. These use greenlets (lightweight threads) allowing a single process to handle hundreds or thousands of concurrent tasks.
pip install gevent
celery -A tasks worker --pool=gevent --concurrency=10002. Prefetch Multiplier
The worker_prefetch_multiplier setting determines how many messages a worker reserves from the broker at once.
- Default (4): Good for tasks with mixed durations.
- Long-running tasks: Set to 1. This prevents one worker from hogging tasks while others sit idle.
- Short, fast tasks: Increase to 10 or more. This reduces the overhead of round-trips to the broker.
# In celery config
app.conf.worker_prefetch_multiplier = 103. Ignoring Results
Storing task results in a backend (Redis/DB) adds significant overhead (network I/O + serialization). If you don't need the return value, disable it.
Global Setting
app.conf.task_ignore_result = TruePer-Task Setting
@app.task(ignore_result=True)
def fire_and_forget_task():
# ...
pass4. Serialization Protocol
JSON is the default and is generally fast, but for extremely high throughput, consider MessagePack or Pickle (if security allows).
- Pickle: Fastest for Python objects, but insecure (don't use if you accept tasks from untrusted sources).
- Msgpack: Binary serialization, faster and smaller than JSON.
pip install msgpackapp.conf.task_serializer = 'msgpack'
app.conf.result_serializer = 'msgpack'
app.conf.accept_content = ['msgpack']5. Connection Pooling
If you are using a database backend (like PostgreSQL) or Redis, ensure connection pooling is enabled and sized correctly so workers don't spend time establishing new connections for every task.
For Redis, Celery handles this well by default, but ensure your Redis server maxclients setting is high enough to handle concurrency * number_of_workers.