Chapter 7. Distributing Tasks with Celery

In the previous chapter, we learned about using parallel Python. We saw the implementation of case studies, including Fibonacci series terms and Web crawler using the parallel Python module. We learned how to establish communication among processes using pipes and how to distribute processes among different machines in a network. In this chapter, we will study how to distribute tasks among different machines in a network by using the Celery framework.

In this chapter, we will cover the following topics:

• Understanding Celery

• Understanding Celery's architecture

• Setting up the environment

• Dispatching a simple task

• Using Celery to obtain a Fibonacci series term

• Using Celery to make a distributed Web crawler