Celery Canvas: Designing Workflows
Celery Canvas is a set of tools used to define complex workflows. It allows you to link tasks together, run them in parallel, or combine them in various ways using Signatures.
1. Signatures (subtasks)
A signature wraps the arguments, keyword arguments, and execution options of a single task invocation in a way that it can be passed to functions or serialized.
from tasks import add
# Create a signature
sig = add.s(2, 2)
# Execute it later
result = sig.delay()
print(result.get()) # 42. Chains (Sequential)
A Chain links tasks together so that one returns after the other. The result of the first task is passed as the first argument to the next task.
from celery import chain
from tasks import add, mul
# (4 + 4) * 8
# add(4, 4) -> 8
# mul(8, 8) -> 64
workflow = chain(add.s(4, 4) | mul.s(8))
result = workflow.delay()
print(result.get())Note: The pipe operator | is syntactic sugar for chain.
3. Groups (Parallel)
A Group executes a list of tasks in parallel. It returns a GroupResult that keeps track of the results of all tasks in the group.
from celery import group
from tasks import add
# Run three add tasks in parallel
job = group(add.s(1, 1), add.s(2, 2), add.s(3, 3))
result = job.delay()
print(result.get())
# Output: [2, 4, 6]4. Chords (Group + Callback)
A Chord allows you to define a callback to be called when all tasks in a group have finished. This is essential for "MapReduce" style workflows.
from celery import chord
from tasks import add, xsum
# 1. Run the group (header)
# 2. When finished, run xsum (body) with the list of results
workflow = chord(
(add.s(i, i) for i in range(10)),
xsum.s()
)
result = workflow.delay()
print(result.get()) # Sum of (0+0, 1+1, ... 9+9)