Discovering Python's parallel programming tools

Exclusive offer: get 50% off this eBook here
Parallel Programming with Python

Parallel Programming with Python — Save 50%

Develop efficient parallel systems using the robust Python environment with this book and ebook.

$13.99    $7.00
by Jan Palach | June 2014 | Open Source

In this article by Jan Palach, author of Parallel Programming with Python, we will expose some Python libraries to implement parallel solutions.

The Python language, created by Guido Van Rossum, is a multi-paradigm, multi-purpose language. It has been widely accepted worldwide due to its powerful simplicity and easy maintenance. It is also known as the language that has batteries included. There is a wide range of modules to make its use smoother. Within parallel programming, Python has built-in and external modules that simplify implementation. This work is based on Python 3.x.

(For more resources related to this topic, see here.)

The Python threading module

The Python threading module offers a layer of abstraction to the module _thread, which is a lower-level module. It provides functions that help the programmer during the hard task of developing parallel systems based on threads. The threading module's official papers can be found at http://docs.python.org/3/library/threading.html?highlight=threading#module-threadin.

The Python multiprocessing module

The multiprocessing module aims at providing a simple API for the use of parallelism based on processes. This module is similar to the threading module, which simplifies alternations between the processes without major difficulties. The approach that is based on processes is very popular within the Python users' community as it is an alternative to answering questions on the use of CPU-Bound threads and GIL present in Python. The multiprocessing module's official papers can be found at http://docs.python.org/3/library/multiprocessing.html?highlight=multiprocessing#multiprocessing.

The parallel Python module

The parallel Python module is external and offers a rich API for the creation of parallel and distributed systems making use of the processes approach. This module promises to be light and easy to install, and integrates with other Python programs. The parallel Python module can be found at http://parallelpython.com. Among some of the features, we may highlight the following:

  • Automatic detection of the optimal confi guration
  • The fact that a number of worker processes can be changed during runtime
  • Dynamic load balance
  • Fault tolerance
  • Auto-discovery of computational resources

Celery – a distributed task queue

Celery is an excellent Python module that's used to create distributed systems and has excellent documentation. It makes use of at least three different types of approach to run tasks in concurrent form—multiprocessing, Eventlet, and Gevent. This work will, however, concentrate efforts on the use of the multiprocessing approach. Also, the link between one and another is a configuration issue, and it remains as a study so that the reader is able to establish comparisons with his/her own experiments.

The Celery module can be obtained on the official project page at http://celeryproject.org.

Summary

In this article, we had a short introduction to some Python modules, built-in and external, which makes a developer's life easier when building up parallel systems.

Resources for Article:


Further resources on this subject:


Parallel Programming with Python Develop efficient parallel systems using the robust Python environment with this book and ebook.
Published: June 2014
eBook Price: $13.99
Book Price: $22.99
See more
Select your format and quantity:

About the Author :


Books From Packt


Practical Maya Programming with Python
Practical Maya Programming with Python

Building probabilistic graphical models with Python
Building probabilistic graphical models with Python

Spring Python 1.1
Spring Python 1.1

MySQL for Python
MySQL for Python

Python Geospatial Development - Second Edition
Python Geospatial Development - Second Edition

Python Data Visualization Cookbook
Python Data Visualization Cookbook

Mastering Python Regular Expressions
Mastering Python Regular Expressions

Learning Geospatial Analysis with Python
Learning Geospatial Analysis with Python


Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software