Using DynamoDB as a Result Backend for Celery
Amazon DynamoDB is a serverless, NoSQL database service. Using it as a result backend for Celery is excellent for scalability and serverless architectures, especially when paired with SQS as the broker.
1. Installation
You need to install the AWS SDK for Python (boto3) alongside Celery.
pip install boto32. Configuration
Configure the result_backend using the dynamodb:// scheme.
URL Format
The format is:
dynamodb://aws_access_key_id:aws_secret_access_key@region/table_name
If you are using IAM Roles (EC2/ECS/Lambda) or environment variables (standard AWS setup), you can omit the credentials:
# Format: dynamodb://@region/table_name
app = Celery('tasks',
broker='sqs://',
backend='dynamodb://@us-east-1/celery-results')3. Table Configuration
Celery can automatically create the table for you if it doesn't exist, but in production, you should provision it via Terraform or CloudFormation to manage capacity modes (On-Demand vs Provisioned).
Schema Requirements
- Partition Key (Hash Key):
id(String)
Configuration Options
You can pass specific DynamoDB settings via result_backend_transport_options.
app.conf.result_backend_transport_options = {
'read_capacity_units': 5,
'write_capacity_units': 5,
}4. Time To Live (TTL)
DynamoDB supports automatic expiration of items. This is crucial for a result backend to prevent the table from growing infinitely.
Celery does not automatically enable TTL on the table definition; you must enable it on the AWS console or via IaC.
- Go to the DynamoDB Table in AWS Console.
- Select Time to Live (TTL).
- Enable it and set the attribute name to
ttl(Celery uses this field name by default for expiration).
5. IAM Permissions
The worker needs permissions like dynamodb:GetItem, dynamodb:PutItem, and dynamodb:DescribeTable on the specific table resource.