Using Celery with Django

Let us try to understand how Celery works under the hood by taking a small real-world example. Imagine you want to perform 10 different tasks and you have 2 people to work on them. Each person can work on only one particular task at a time, and new tasks might arrive at any given time. To solve this problem, the workers would create a list, and every time a new task comes in, they would add the task to that list. Each time a person finishes their task, you would strike off the task from the list and then assign them the next task from the top. If you add more people to work for you, then you can finish your tasks faster. This is what Celery does.

In Celery, there is a task queue that keeps track of all the tasks coming in – task queues are saved using RabbitMQ or Redis. There are Celery workers that pick tasks from the queue and work on them. Django generates these tasks and pushes them to the task queue, from where Celery picks tasks and executes...