Using Amazon SQS as a Broker for Celery
Amazon Simple Queue Service (SQS) is a fully managed message queuing service. Using SQS as a Celery broker is a great choice if you are already running your infrastructure on AWS, as it removes the need to manage a RabbitMQ or Redis server.
1. Installation
You need to install Celery with the SQS bundle.
pip install "celery[sqs]"2. Configuration
To use SQS, you configure the broker_url using the sqs:// scheme.
Broker URL Format
The basic format is sqs://aws_access_key_id:aws_secret_access_key@.
However, it is highly recommended to use IAM roles or environment variables instead of hardcoding credentials in the URL. If your environment (e.g., EC2, ECS, local env vars) is configured with AWS credentials, you can simply use:
app = Celery('tasks', broker='sqs://')3. Transport Options
SQS behaves differently than RabbitMQ or Redis. You must configure broker_transport_options to handle regions and timeouts correctly.
app.conf.broker_transport_options = {
'region': 'us-east-1',
'visibility_timeout': 3600, # 1 hour
'polling_interval': 1, # Check for messages every 1 second
}region: The AWS region where your queues are located.visibility_timeout: The time (in seconds) a task remains invisible to other workers after being picked up. Crucial: If your task takes longer than this to execute, SQS will assume the worker died and redeliver the message to another worker, causing the task to run twice. Set this value higher than your longest expected task duration.polling_interval: How often the worker polls SQS for new messages.
4. Important Considerations
No Result Backend
SQS is strictly a message broker; it cannot be used to store results. If you need to check the return value of tasks (result.get()), you must configure a separate backend (e.g., Redis, DynamoDB, or SQLAlchemy).
app = Celery('tasks',
broker='sqs://',
backend='redis://localhost:6379/0')Permissions
The AWS credentials used by the worker must have permissions to perform actions on SQS, specifically:
sqs:SendMessagesqs:ReceiveMessagesqs:DeleteMessagesqs:GetQueueUrlsqs:CreateQueue(if you want Celery to auto-create queues)
Limitations
SQS does not support Pub/Sub mechanisms natively. This means Celery's remote control commands (like inspecting workers or revoking tasks via broadcast) are limited or require specific workarounds compared to RabbitMQ or Redis.