Deploy a Python Celery Worker10 min
Celery is an open-source distributed task queue that focuses on real-time processing and task scheduling. It coordinates tasks through a backing message broker, typically Redis or RabbitMQ, making it straightforward to scale as demand increases. While Celery is written in Python, its messaging protocols are not language specific, allowing for clients in any language.
In this guide, we will go over how to deploy a Celery worker on the Koyeb serverless platform using git-driven deployment. We will create a repository with Celery and its dependencies installed, deploy RabbitMQ to act as our backing message broker, and then connect our GitHub account to Koyeb to automatically deploy the Celery worker to the platform.
At the end of this guide, you should have a Celery worker deployed to Koyeb that you can automatically update by pushing changes to your GitHub account.
To follow along with this guide, you will need the following:
- Python 3 installed on your local machine
- A Koyeb account to deploy and run your Celery worker and RabbitMQ message broker
- A GitHub account to store your application code and trigger deployments via repository changes
This guide will cover how to deploy a Celery worker to Koyeb through the following steps:
- Create a Python Virtual Environment
- Install Celery
- Create a Basic Celery Application
- Push the Celery Project to GitHub
- Deploy RabbitMQ on Koyeb
- Deploy the Celery Worker on Koyeb
- Testing the Celery Worker
Create a Python Virtual Environment
To get started, we will create a Python virtual environment on our local machine to isolate our project's dependencies from the rest of the system packages. Python provides this functionality natively through the standard
Start by creating a new directory to store our project files.
Next, create the virtual environment within the project directory by typing:
This will create a directory called
venv within the
~/celery-koyeb containing initialization scripts, copies of essential tools like
pip, and links to the local Python interpreters.
To tell Python to use the virtual environment, activate it by sourcing the initialization script:
Your shell prompt will change to reflect that the virtual environment is active for this session. Python will now use this directory structure in place of the system's standard Python file locations, allowing us to isolate dependencies.
Next, we need to install Celery within the virtual environment using
pip, the Python package manager:
Python will download Celery and its dependencies and install them within the virtual environment filesystem.
Along with the Python modules, the installation also includes a small wrapper script called
celery that lets you interact with it from the command line. Use the command now to verify that the installation was successful:
The command should show the version as expected. The exact version returned does not matter, so long as the command returns successfully:
With Celery installed, record the project's dependencies by typing:
Create a Basic Celery Application
Next, we'll create a basic Celery application in our project directory.
Create a file called
tasks.py within the
~/celery-koyeb directory with the following content inside:
Let's talk a bit about what is happening in the code above.
After importing the
os module and the
Celery class, the code above sets the location of the backing RabbitMQ message broker based on the value of the
BROKER_HOST environment variable. If this variable is not set, we will error out to avoid accidentally deploying misconfigured instances.
Next, we create an instance of the
Celery class called
app. We pass
tasks as the name of the main module and include the broker and backend configuration to allow Celery to connect to the RabbitMQ instance indicated by the environment variable mentioned earlier.
Finally, we create a function with the
@app.task decorator to add it to Celery's task registry. Our function adds two numbers together and returns the result.
Save and close the file when you are finished.
Push the Celery Project to GitHub
Next, we will commit our project to
git and push our changes to our GitHub account.
In your project directory, initialize a
git repository to get started:
tasks.py application and the
requirements.txt file to the staging area. You can also create a minimal
.gitignore file to make sure your virtual environment does not accidentally get committed. Commit the changes when you are done:
Create a new repository within your GitHub account. This can be either public or private. Add the GitHub repository address as the
origin of your project and push your changes to the repository:
If you refresh the GitHub repository page in your browser, your project files will appear.
Deploy RabbitMQ on Koyeb
Before we deploy our Celery worker, we need to deploy a RabbitMQ instance for Celery to use as a message broker. We will use the official RabbitMQ image on DockerHub for this purpose.
Go to the Koyeb control panel and click the Create App button. Select Docker as the deployment method to continue.
On the next screen, in the Image field, type
docker.io/rabbitmq and click Next.
Next, choose Web Service as your service type and select your desired region and instance size. Click Advanced to expand the networking configuration options.
In the Environment variables section, change the value of the
PORT variable to RabbitMQ's default port: 5672.
In the Exposing your service section, change the port number to 5672 and deselect the Public option. This will allow other services within this application (like our Celery application) to access the message broker using its service name without exposing the service to the internet.
When you have completed the above configuration, click the Deploy button.
Your new application will be created and the RabbitMQ service will begin to deploy within it. In a few minutes, the RabbitMQ container will finish deploying to Koyeb and the service will be marked as healthy.
Before moving on, copy the Private domain of your RabbitMQ deployment. We will need this later as we configure the Celery worker service.
Deploy the Celery Worker on Koyeb
Now that the RabbitMQ service is deployed, we can deploy the Celery application from our GitHub repository.
In the Koyeb control panel, click the app that contains your RabbitMQ service. Click the Create Service button to deploy a new service to the app alongside the RabbitMQ container. Select GitHub as the deployment method to continue.
Note: If this is your first time using Koyeb with GitHub, click the Install GitHub App button to connect your accounts. On the authorization page, select which repositories you want Koyeb to access and click the Install button to continue. You will be redirected back to Koyeb to continue your deployment.
Next, select the GitHub repository for your Celery application to continue.
On the following page, click Build and deployment settings to expand the options for your deployment. Click the Override toggle associated with the Run command field so that we can specify the command we want to run in this deployment. In the Run command field, type:
This will start a Celery worker using the
app instance we configured within the
Next, select Worker as the service type and select your desired region and instance size. Click Advanced to expand additional configuration options.
In the Environment variables section, create a new variable called
BROKER_HOST to tell the Celery worker where to find the RabbitMQ instance. Set the value to the Private domain you copied from the RabbitMQ service. It will follow this format:
When you have completed the above steps, click the Deploy button.
The Celery application will be pulled from your GitHub repository and deployed. In a few minutes, the Celery worker will begin running within the service and be available for work.
If you want to learn about how Koyeb automatically builds your Python applications from git, make sure to read the how we build from git documentation.
Testing the Celery Worker
Unlike a web application, Celery does not have a web interface to easily show you that the deployment succeeded. We can, however, use Koyeb's built-in Console to test that the worker is running, able to connect with RabbitMQ, and can process tasks.
In the Koyeb control panel, click the Celery service and then select the Console tab.
In the console window, start a
bash session by typing:
Next, source the environment configuration file so that Python can find the modules associated with our application:
Start a Python session so that we can interact with the Celery application:
First, import our
Next, we can run the
add function using Celery's
We can then use the
results object to check on the task's progress. Call the
ready method to see if it has been processed yet:
You can check the answer returned by the function using the
get method. It's usually a good idea to set a timeout so that it won't hang waiting if the result is not available yet:
While this test uses trivial calculations, the results validate that the system is working as intended and able to communicate with RabbitMQ using Koyeb's service mesh.
When you are finished, exit the Python session by typing:
You can then close the bash session by typing:
In this guide, we covered how to deploy a Celery worker on Koyeb using git-driven deployment. Because Celery requires a message broker for coordination, we also discussed how to deploy a RabbitMQ container on Koyeb using Docker-based deployment.
By deploying both of these services within a single application, we were able to configure Celery to connect with RabbitMQ over Koyeb's service mesh with minimal configuration. Our Celery deployment is backed by a GitHub repository, so it will automatically redeploy whenever new changes are pushed to the branch it is tracking. This makes it easy to change your running Celery worker configuration by modifying your project's code.
Questions or suggestions to improve this guide? Join us on the community platform to chat!