Deploying Celery on AWS Elastic Beanstalk
AWS Elastic Beanstalk (EB) makes it easy to deploy, manage, and scale applications. Running background tasks with Celery on EB typically involves one of two architectures: running the worker alongside the web server or using a dedicated worker environment.
Approach 1: The Procfile Method (Recommended for Amazon Linux 2/2023)
Modern Elastic Beanstalk platforms (Amazon Linux 2 and newer) use a Procfile to manage processes. This is the simplest way to run a Celery worker alongside your web application.
1. Create a Procfile
Create a file named Procfile (no extension) in the root of your project source code.
web: gunicorn myproject.wsgi:application
worker: celery -A myproject worker --loglevel=INFO
beat: celery -A myproject beat --loglevel=INFO- web: Your main application server (e.g., Gunicorn for Django).
- worker: The Celery worker process.
- beat: (Optional) The Celery Beat scheduler. Note: Only run one Beat instance per deployment to avoid duplicate tasks.
2. How it Works
Elastic Beanstalk reads this file and uses the process manager (usually Systemd) to keep these processes running. If the worker crashes, EB restarts it. Logs are aggregated automatically.
Approach 2: Dedicated Worker Tier
For high-load applications, you might want to separate your web instances from your worker instances. EB offers a specific "Worker Environment" tier.
The SQS Daemon (sqsd) vs. Celery
By default, the EB Worker Tier runs a daemon called sqsd. It polls an SQS queue and sends an HTTP POST request to your application (e.g., /tasks/process) with the message body.
- If you use
sqsd: You don't strictly need Celery; you just need an endpoint in your Flask/Django app to handle the POST request. - If you want standard Celery: You can disable
sqsdand use theProcfilemethod within the Worker Tier to runcelery workerconsuming from SQS (or Redis).
Configuration Tips
1. Environment Variables
You must configure your broker URL via the EB Console or CLI.
eb setenv CELERY_BROKER_URL="redis://my-elasticache-endpoint:6379/0"2. Installing System Dependencies
If your tasks require system libraries (like libpq-dev or Pillow dependencies), use .ebextensions.
Create .ebextensions/01_packages.config:
packages:
yum:
postgresql-devel: []
libjpeg-turbo-devel: []3. Deployment Hooks (Advanced)
If you need to run scripts before the worker starts (e.g., to clean up PID files), use .platform/hooks.
File: .platform/hooks/prebuild/01_cleanup.sh
#!/bin/bash
rm -f celerybeat.pid