Using Celery with Django

We have learned about the Django and DRF features that are provided out of the box in our previous chapters. Django and DRF both provide a scope of customization and extendibility to the framework, but async programming has been missing from Django for quite some time. However, with the introduction of Django 4, Django has started to natively support async programming using different async interfaces. This still doesn’t help with requirements to execute long-running jobs or jobs executed at periodic intervals. To address this requirement, we use a well-known third-party package in Python, Celery.

Celery is a simple, flexible, and reliable distributed system used to process vast amounts of messages. It is a task queue with focus on real-time processing while also supporting task scheduling. In this chapter, we shall learn how to set up Celery with Django and use celery beat to perform periodic tasks in Django. Since Celery itself is an altogether different...

Technical requirements

In this chapter, we will cover some advanced concepts of asynchronous programming, so it is important that the readers are well versed in the following:

• The difference between synchronous and asynchronous programming, and the advantages of the latter over the former.
• The basic concepts of messaging brokers and task queues. Celery is a framework used to process task queues and task scheduling.
• How to set up Redis and use a remote Redis server in local system (we discussed this in Chapter 6).

Asynchronous programming in Django

Django 4 introduced the concept of asynchronous programming in its core framework. While support for async views is introduced in Django, the official documentation (https://docs.djangoproject.com/en/stable/topics/async/#performance) states that async views would lead to a performance hit.

Hence, the adaption of native asynchronous programming is not yet mature as of Django 4.x. While most common and basic features will not need any async support, Django as a framework is thus lagging behind other frameworks such as Express and Spring. To work around this problem, Django developers use third-party frameworks to implement asynchronous support. Generally, asynchronous programming is helpful when one has a long-running task that can be executed in the background. For example, when we have to send an email, our Django view would make an external API call to an email service and await the response. This whole process can take a while and there is no...

Using Celery with Django

Let us try to understand how Celery works under the hood by taking a small real-world example. Imagine you want to perform 10 different tasks and you have 2 people to work on them. Each person can work on only one particular task at a time, and new tasks might arrive at any given time. To solve this problem, the workers would create a list, and every time a new task comes in, they would add the task to that list. Each time a person finishes their task, you would strike off the task from the list and then assign them the next task from the top. If you add more people to work for you, then you can finish your tasks faster. This is what Celery does.

In Celery, there is a task queue that keeps track of all the tasks coming in – task queues are saved using RabbitMQ or Redis. There are Celery workers that pick tasks from the queue and work on them. Django generates these tasks and pushes them to the task queue, from where Celery picks tasks and executes...

Using celery beat with Django

Periodic tasks are one of the most common use cases for Celery. For example, say you want to send periodic emails to your users every week, or create certain summary reports every week. For such repetitive tasks, Celery provides celery beat, which is a scheduler that keeps checking for any scheduled task to be executed at the given time and then runs it. The celery beat worker would create tasks in place of the Django application and put them in the Redis task queue, from where tasks are then picked up by the Celery workers and executed.

Let us learn how to integrate celery beat into our Django project:

1. There are multiple ways to create periodic tasks, such as adding a crontab configuration in the settings file, or plugging a beat schedule into the app.conf.beat_schedule object. They work perfectly, but lack ease of use. One option that truly stands out is using the on_after_finalize.connect signal.

   In the following example, we create a periodic...

Summary

In this chapter, we learned how to tackle the problem of long-running tasks using Celery. In web development, having a long-running task is quite common, such as when we have to generate a monthly report, send an email to a batch of users, process data from an uploaded file, and so on. Celery is very handy to address such use cases. Similarly, there are plenty of use cases where you need task-scheduling functionality. For example, if we wanted to send a weekly consolidated report to users every week, send reminder emails to end users, and so on, celery beat would be our go-to tool to build such features.

In Chapter 9, we will learn how to write unit and integration tests with Django. Writing unit tests is one of the most important yet frequently overlooked phase in the development cycle. We will learn all the best practices to follow while writing unit and integration tests.