Using Amazon S3 as a Result Backend for Celery
When dealing with tasks that return large data payloads (like generated reports, images, or large datasets), storing results in Redis or a database can be inefficient or expensive. Amazon S3 is an excellent alternative for storing these large result blobs due to its low cost and high durability.
1. Installation
You need to install Celery with the S3 bundle, which includes boto3.
pip install "celery[s3]"2. Configuration
While you can encode credentials directly into the result_backend URL, it is cleaner and more secure to use result_backend_transport_options, especially when using IAM roles.
Basic Setup (IAM Roles / Env Vars)
If your environment (EC2, ECS, Lambda, Local) already has AWS credentials configured via environment variables or IAM roles, you can keep the configuration simple.
app = Celery('tasks',
broker='redis://localhost:6379/0',
# The URL scheme triggers the S3 backend
backend='s3://')
# Configure S3 specifics here
app.conf.result_backend_transport_options = {
'bucket': 'my-celery-results-bucket',
'region': 'us-east-1',
}Explicit Credentials (Not Recommended)
If you must hardcode credentials (e.g., for local testing without env vars), you can pass them in the options:
app.conf.result_backend_transport_options = {
'aws_access_key_id': 'YOUR_KEY',
'aws_secret_access_key': 'YOUR_SECRET',
'bucket': 'my-celery-results-bucket',
'region': 'us-east-1',
}3. S3 Bucket Configuration
- Create the Bucket: Ensure the bucket defined in your configuration exists.
- Lifecycle Policy: Celery does not automatically delete old results from S3 (unlike Redis keys with a TTL). You must configure an S3 Lifecycle Rule in the AWS Console to expire/delete objects after a certain period (e.g., 7 days) to prevent storage costs from growing indefinitely.
4. When to use S3?
| Feature | Redis/Memcached | Database | Amazon S3 |
|---|---|---|---|
| Speed | Very Fast | Fast | Slower (HTTP latency) |
| Payload Size | Small (< 1MB) | Medium | Large (GBs) |
| Cost | High (RAM) | Medium | Low (Storage) |
| Persistence | Volatile | Durable | Durable |
Recommendation: Use S3 if your tasks return generated files, PDFs, or large JSON blobs. Use Redis if your tasks return simple status codes, small strings, or database IDs.
Note: Reading results from S3 is slower than Redis. Ensure your application can tolerate the latency of an HTTP request when fetching result.get().