# Using Amazon S3 as a Result Backend for Celery

When dealing with tasks that return large data payloads (like generated reports, images, or large datasets), storing results in Redis or a database can be inefficient or expensive. Amazon S3 is an excellent alternative for storing these large result blobs due to its low cost and high durability.

## 1. Installation

You need to install Celery with the S3 bundle, which includes `boto3`.

```bash
pip install "celery[s3]"
```

## 2. Configuration

While you can encode credentials directly into the `result_backend` URL, it is cleaner and more secure to use `result_backend_transport_options`, especially when using IAM roles.

### Basic Setup (IAM Roles / Env Vars)

If your environment (EC2, ECS, Lambda, Local) already has AWS credentials configured via environment variables or IAM roles, you can keep the configuration simple.

```python
app = Celery('tasks',
             broker='redis://localhost:6379/0',
             # The URL scheme triggers the S3 backend
             backend='s3://')

# Configure S3 specifics here
app.conf.result_backend_transport_options = {
    'bucket': 'my-celery-results-bucket',
    'region': 'us-east-1',
}
```

### Explicit Credentials (Not Recommended)

If you must hardcode credentials (e.g., for local testing without env vars), you can pass them in the options:

```python
app.conf.result_backend_transport_options = {
    'aws_access_key_id': 'YOUR_KEY',
    'aws_secret_access_key': 'YOUR_SECRET',
    'bucket': 'my-celery-results-bucket',
    'region': 'us-east-1',
}
```

## 3. S3 Bucket Configuration

1.  **Create the Bucket**: Ensure the bucket defined in your configuration exists.
2.  **Lifecycle Policy**: Celery **does not** automatically delete old results from S3 (unlike Redis keys with a TTL). You must configure an **S3 Lifecycle Rule** in the AWS Console to expire/delete objects after a certain period (e.g., 7 days) to prevent storage costs from growing indefinitely.

## 4. When to use S3?

| Feature | Redis/Memcached | Database | Amazon S3 |
| :--- | :--- | :--- | :--- |
| **Speed** | Very Fast | Fast | Slower (HTTP latency) |
| **Payload Size** | Small (< 1MB) | Medium | Large (GBs) |
| **Cost** | High (RAM) | Medium | Low (Storage) |
| **Persistence** | Volatile | Durable | Durable |

**Recommendation:** Use S3 if your tasks return generated files, PDFs, or large JSON blobs. Use Redis if your tasks return simple status codes, small strings, or database IDs.

*Note: Reading results from S3 is slower than Redis. Ensure your application can tolerate the latency of an HTTP request when fetching `result.get()`.*

[[programming/python/celery]]