Introducing task queues

Producers and workers are independent and need a mechanism for producers to offload tasks to workers and, occasionally, for workers to send back a response to a producer. The microservices architecture uses task queues for this communication. A task queue is what its name implies: a queue where producers place tasks and workers pick up those tasks. In its most basic form, a task queue operates like this:

Figure 6.1 – A basic task queue

Besides this basic scenario, task queue managers support more complex scenarios such as the following:

• Work queues, where multiple workers share the tasks
• Publish-Subscribe, where multiple workers receive and process all tasks
• Request-Response, where workers send back a response to the producer

Let’s look at how these scenarios work, including code examples based on the pika package for the producers and workers and with RabbitMQ as the task queue manager. We&...