What do you get with Print?

Instant access to your digital copy whilst your Print order is Shipped

Paperback book shipped to your preferred address

Redeem a companion digital copy on all Print orders

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

Parallel Programming with Python

Chapter 1. Contextualizing Parallel, Concurrent, and Distributed Programming

Parallel programming can be defined as a model that aims to create programs that are compatible with environments prepared to execute code instructions simultaneously. It has not been too long since techniques of parallelism began to be used to develop software. Some years ago, processors had a single Arithmetic Logic Unit (ALU) among other components, which could only execute one instruction at a time during a time space. For years, only a clock that measured in hertz to determine the number of instructions a processor could process within a given interval of time was taken into consideration. The more the number of clocks, the more the instructions potentially executed in terms of KHz (thousands of operations per second), MHz (millions of operations per second), and the current GHz (billions of operations per second).

Summing up, the more instructions per cycle given to the processor, the faster the execution. During the '80s, a revolutionary processor came to life, Intel 80386, which allowed the execution of tasks in a pre-emptive manner, that is, it was possible to periodically interrupt the execution of a program to provide processor time to another program; this meant pseudo-parallelism based on time-slicing.

In the late '80s, there came Intel 80486 that implemented a pipelining system, which in practice, divided the stage of execution into distinct substages. In practical terms, in a cycle of the processor, we could have different instructions being carried out simultaneously in each substage.

All the advances mentioned in the preceding section resulted in several improvements in performance, but it was not enough, as we were faced with a delicate issue that would end up as the so-called Moore's law (http://www.mooreslaw.org/).

The quest for high taxes of clock ended up colliding with physical limitations; processors would consume more energy, thereby generating more heat. Moreover, there was another as important issue: the market for portable computers was speeding up in the '90s. So, it was extremely important to have processors that could make the batteries of these pieces of equipment last long enough away from the plug. Several technologies and families of processors from different manufacturers were born. As regards servers and mainframes, Intel® deserves to be highlighted with its family of products Core®, which allowed to trick the operating system by simulating the existence of more than one processor even though there was a single physical chip.

In the Core® family, the processor got severe internal changes and featured components called core, which had their own ALU and caches L2 and L3, among other elements to carry out instructions. Those cores, also known as logical processors, allowed us to parallel the execution of different parts of the same program, or even different programs, simultaneously. The age core enabled lower energy use with power processing superior to its predecessors. As cores work in parallel, simulating independent processors, we can have a multi-core chip and an inferior clock, thereby getting superior performance compared to a single-core chip with higher clock, depending on the task.

So much evolution has, of course, changed the way we approach software designing. Today, we must think of parallelism to design systems that make rational use of resources without wasting them, thereby providing a better experience to the user and saving energy not only in personal computers, but also at processing centers. More than ever, parallel programming is in the developers' daily lives and, apparently, it will never go back.

This chapter covers the following topics:

Why use parallel programming?
Introducing the common forms of parallelization
Communicating in parallel programming
Identifying parallel programming problems
Discovering Python's programming tools
Taking care of Python Global Interpreter Lock (GIL)

Exploring common forms of parallelization

There is a certain confusion when we try to define the main forms of paralleling systems. It is common to find quotations on parallel and concurrent systems as if both meant the same thing. Nevertheless, there are slight differences between them.

Within concurrent programming, we have a scenario in which a program dispatches several workers and these workers dispute to use the CPU to run a task. The stage at which the dispute takes place is controlled by the CPU scheduler, whose function is to define which worker is apt for using the resource at a specific moment. In most cases, the CPU scheduler runs the task of raking processes so fast that we might get the impression of pseudo-parallelism. Therefore, concurrent programming is an abstraction from parallel programming.

Note

Concurrent systems dispute over the same CPU to run tasks.

The following diagram shows a concurrent program scheme:

Exploring common forms of parallelization

Concurrent programming scheme.

Parallel programming can be defined as an approach in which program data creates workers to run specific tasks simultaneously in a multicore environment without the need for concurrency amongst them to access a CPU.

Note

Parallel systems run tasks simultaneously.

The following figure shows the concept of parallel systems:

Parallel programming scheme.

Distributed programming aims at the possibility of sharing the processing by exchanging data through messages between machines (nodes) of computing, which are physically separated.

Distributed programming is becoming more and more popular for many reasons; they are explored as follows:

Fault-tolerance: As the system is decentralized, we can distribute the processing to different machines in a network, and thus perform individual maintenance of specific machines without affecting the functioning of the system as a whole.
Horizontal scalability: We can increase the capacity of processing in distributed systems in general. We can link new equipment with no need to abort applications being executed. We can say that it is cheaper and simpler compared to vertical scalability.
Cloud computing: With the reduction in hardware costs, we need the growth of this type of business where we can obtaining huge machine parks acting in a cooperative way and running programs in a transparent way for their users.

Note

Distributed systems run tasks within physically-separated nodes.

The following figure shows a distributed system scheme:

Distributed programming scheme.

Communicating in parallel programming

In parallel programming, the workers that are sent to perform a task often need to establish communication so that there can be cooperation in tackling a problem. In most cases, this communication is established in such a way that data can be exchanged amongst workers. There are two forms of communication that are more widely known when it comes to parallel programming: shared state and message passing. In the following sections, a brief description of both will be presented.

Understanding shared state

One the most well-known forms of communication amongst workers is shared state. Shared state seems straightforward to use but has many pitfalls because an invalid operation made to the shared resource by one of the processes will affect all of the others, thereby producing bad results. It also makes it impossible for the program to be distributed between multiple machines for obvious reasons.

Illustrating this, we will make use of a real-world case. Suppose you are a customer of a specific bank, and this bank has only one cashier. When you go to the bank, you must head to a queue and wait for your chance. Once in the queue, you notice that only one customer can make use of the cashier at a time, and it would be impossible for the cashier to attend two customers simultaneously without potentially making errors. Computing provides means to access data in a controlled way, and there are several techniques, such as mutex.

Mutex can be understood as a special process variable that indicates the level of availability to access data. That is, in our real-life example, the customer has a number, and at a specific moment, this number will be activated and the cashier will be available for this customer exclusively. At the end of the process, this customer will free the cashier for the next customer, and so on.

Note

There are cases in which data has a constant value in a variable while the program is running, and the data is shared only for reading purposes. So, access control is not necessary because it will never present integrity problems.

Understanding message passing

Message passing is used when we aim to avoid data access control and synchronizing problems originating from shared state. Message passing consists of a mechanism for message exchange in running processes. It is very commonly used whenever we are developing programs with distributed architecture, where the message exchanges within the network they are placed are necessary. Languages such as Erlang, for instance, use this model to implement communication in its parallel architecture. Once data is copied at each message exchange, it is impossible that problems occur in terms of concurrence of access. Although memory use seems to be higher than in shared memory state, there are advantages to the use of this model. They are as follows:

Absence of data access concurrence
Messages can be exchange locally (various processes) or in distributed environments
This makes it less likely that scalability issues occur and enables interoperability of different systems
In general, it is easy to maintain according to programmers

Identifying parallel programming problems

There are classic problems that brave keyboard warriors can face while battling in the lands where parallel programming ghosts dwell. Many of these problems occur more often when inexperienced programmers make use of workers combined with shared state. Some of these issues will be described in the following sections.

Deadlock

Deadlock is a situation in which two or more workers keep indefinitely waiting for the freeing of a resource, which is blocked by a worker of the same group for some reason. For a better understanding, we will use another real-life case. Imagine the bank whose entrance has a rotating door. Customer A heads to the side, which will allow him to enter the bank, while customer B tries to exit the bank by using the entrance side of this rotating door so that both customers would be stuck forcing the door but heading nowhere. This situation would be hilarious in real life but tragic in programming.

Note

Deadlock is a phenomenon in which processes wait for a condition to free their tasks, but this condition will never occur.

Starvation

This is the issue whose side effects are caused by unfair raking of one or more processes that take much more time to run a task. Imagine a group of processes, A, which runs heavy tasks and has data processor priority. Now, imagine that a process A with high priority constantly consumes the CPU, while a lower priority process B never gets the chance. Hence, one can say that process B is starving for CPU cycles.

Note

Starvation is caused by badly adjusted policies of process ranking.

Race conditions

When the result of a process depends on a sequence of facts, and this sequence is broken due to the lack of synchronizing mechanisms, we face race conditions. They result from problems that are extremely difficult to filter in larger systems. For instance, a couple has a joint account; the initial balance before operations is $100. The following table shows the regular case, in which there are mechanisms of protection and the sequence of expected facts, as well as the result:

Husband	Wife	Account balance (dollars)
		100
Read balance		100
Adds 20		100
Concludes operation		120
	Read balance	120
	Withdraws 10	120
	Concludes operation	110

Presents baking operations without the chance of race conditions occurrence

In the following table, the problematic scenario is presented. Suppose that the account does not have mechanisms of synchronization and the order of operations is not as expected.

Husband	Wife	Account balance (dollars)
		100
Read balance		100
Withdraws 100		100
	Reads balance	100
	Withdraws 10	100
Concludes operation updating balance		0
	Concludes operation updating balance	90

Analogy to balance the problem in a joint account and race conditions

There is a noticeable inconsistency in the final result due to the unexpected lack of synchronization in the operations sequence. One of the parallel programming characteristics is non-determinism. It is impossible to foresee the moment at which two workers will be running, or even which of them will run first. Therefore, synchronization mechanisms are essential.

Note

Non-determinism, if combined with lack of synchronization mechanisms, may lead to race condition issues.

Discovering Python's parallel programming tools

The Python language, created by Guido Van Rossum, is a multi-paradigm, multi-purpose language. It has been widely accepted worldwide due to its powerful simplicity and easy maintenance. It is also known as the language that has batteries included. There is a wide range of modules to make its use smoother. Within parallel programming, Python has built-in and external modules that simplify implementation. This work is based on Python 3.x.

The Python threading module

The Python threading module offers a layer of abstraction to the module _thread, which is a lower-level module. It provides functions that help the programmer during the hard task of developing parallel systems based on threads. The threading module's official papers can be found at http://docs.python.org/3/library/threading.html?highlight=threading#module-threadin.

The Python multiprocessing module

The multiprocessing module aims at providing a simple API for the use of parallelism based on processes. This module is similar to the threading module, which simplifies alternations between the processes without major difficulties. The approach that is based on processes is very popular within the Python users' community as it is an alternative to answering questions on the use of CPU-Bound threads and GIL present in Python. The multiprocessing module's official papers can be found at http://docs.python.org/3/library/multiprocessing.html?highlight=multiprocessing#multiprocessing.

The parallel Python module

The parallel Python module is external and offers a rich API for the creation of parallel and distributed systems making use of the processes approach. This module promises to be light and easy to install, and integrates with other Python programs. The parallel Python module can be found at http://parallelpython.com. Among some of the features, we may highlight the following:

Automatic detection of the optimal configuration
The fact that a number of worker processes can be changed during runtime
Dynamic load balance
Fault tolerance
Auto-discovery of computational resources

Celery – a distributed task queue

Celery is an excellent Python module that's used to create distributed systems and has excellent documentation. It makes use of at least three different types of approach to run tasks in concurrent form—multiprocessing, Eventlet, and Gevent. This work will, however, concentrate efforts on the use of the multiprocessing approach. Also, the link between one and another is a configuration issue, and it remains as a study so that the reader is able to establish comparisons with his/her own experiments.

The Celery module can be obtained on the official project page at http://celeryproject.org.

Description

Starting with the basics of parallel programming, you will proceed to learn about how to build parallel algorithms and their implementation. You will then gain the expertise to evaluate problem domains, identify if a particular problem can be parallelized, and how to use the Threading and Multiprocessor modules in Python. The Python Parallel (PP) module, which is another mechanism for parallel programming, is covered in depth to help you optimize the usage of PP. You will also delve into using Celery to perform distributed tasks efficiently and easily. Furthermore, you will learn about asynchronous I/O using the asyncio module. Finally, by the end of this book you will acquire an in-depth understanding about what the Python language has to offer in terms of built-in and external modules for an effective implementation of Parallel Programming. This is a definitive guide that will teach you everything you need to know to develop and maintain high-performance parallel computing systems using the feature-rich Python.

What you will learn

Explore techniques to parallelize problems

Integrate the Parallel Python module to implement Python code

Execute parallel solutions on simple problems

Achieve communication between processes using Pipe and Queue

Use Celery Distributed Task Queue

Implement asynchronous I/O using the Python asyncio module

Create threadsafe structures

What do you get with Print?

Instant access to your digital copy whilst your Print order is Shipped

Paperback book shipped to your preferred address

Redeem a companion digital copy on all Print orders

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

Frequently bought together

$24.99

$26.99

$48.99

Total $ 100.97

Filter reviews by

All

Amazon verified reviews

Fitzgerald Stowers Aug 04, 2014

Great book and very easy to understand!

Amazon Verified review

Pedro Medeiros Jan 30, 2015

Very useful information. Could be better written, not suitable for parallel programming neophytes.

Zippy Aug 15, 2014

Python’s roots go so far back in time that, despite the prevalence of today’s multicore CPU computers, the most popular, “official” version of the language CPython) is limited by the GIL, or Global Interpreter Lock. The GIL restricts thread processing so that only one thread at a time for each interpreter process executing on the system.Among other things, the GIL prevents the sharing of code that’s not thread-safe with other threads, such as the code often found in the many C libraries that are often called upon by Python programmers, and it allows for simplified memory management.The GIL certainly puts the kibosh on anyone attempting to implement concurrency or pure parallelism on a multiprocessing machine running Python. What one gets instead is a sort of cooperative multitasking of sections of threads that run in spurts and stretches between moments of I/O (read, write, send, etc.) when the GIL is briefly released (CPU-bound threads that never deal with I/O are handled as a special case).Running a Python program on a multiple core PC is therefore asking for trouble, since each core “wants” to schedule its own runnable thread, and all the scheduling of threads is simultaneous. But the GIL only allows one thread at a time to run, so a Python programmer who divvies his algorithm into multiple threads will get quite a surprise when he discovers that his software is running slower than it should, since threads are competing for exclusive access to the GIL. Indeed, running such a multi-thread program on a multicore system can slow things down even more!The limitations imposed by GIL is probably the principal reason why some Python programmers have switched to working in Google’s Go language, which is built from the ground up to harness to power of multiple cores and processors via what its originators call “goroutines;” these vaguely resemble Unix pipes and can all run in the same memory address space via multiplexing onto multiple, parallel OS threads.But all is not lost for those Python programmers longing to delve into the realm of parallel and concurrent programming.For those completely clueless about how to thumb one’s nose at the GIL and tackle concurrency and parallelism in Python, such as using the open source and cross-platform “PP” or Parallel Python module, a good place to start is this book.That being the case, I won’t burden the reader with the quality of the author’s coverage in specific areas, since a beginner will have no idea of what I’m talking about. (Besides, others have commented on that elsewhere on this site.) Suffice it to say that, provided you can jump through enough coding hoops, you can bring a performance boost to your software, particularly if it has a sufficiently “granular” structure.

A. Zubarev Aug 07, 2014

Parallel Programming is an increasingly hot topic in today's IT circles. For those who ponder why I can tell in short it is because of the CPU clock speeds stagnation. We, software engineers, are dealing with ever increasing volumes of data and are asked to deliver even faster, more robust applications and websites. This is tough. Parallel Programming is the answer. I hope I whet your appetite for exploring the Parallel Programming so now I can switch the focus to the book.It is not terribly long. Not costly either. In fact if you care I managed to read it whole in 3 hours plus (stats are from my ebook reader app) and managed to run a few examples that worked on my laptop with Windows 7. I am planning on running more examples later on a POSIX machine. Thing with the examples is they are classic ones: the Fibonacci series which is boring to me and far from what anybody would be dealing with at work and web crawling which is better done using say Nutch. The same code examples go through the entire book, just different techniques applied. What I wish Jan had done is explaining at least what technique helps in what case in real life. My other pet peeves are that there was no mention on how to leverage the GPU, how to eliminate the For Loops - this is actually a must in my opinion, and there was no coverage on how to debug parralel processes. Let me stop at debugging a tad longer: since Python allows mutability it becomes critical to exterminate nasty mutation bugs!In terms of closing, I have an advice to the author: it is hard to write a technical book, but I wish it could be longer and covered more ground, another advice is to the publisher, this book qualifies for the "Instant" moniker type of the books from Packt. By the way I like your website redesign!Three stars out of five.

Brisard Aug 14, 2014

For a long time, programmers have been relying exclusively on Moore's law to resolve their performance issues. In other words, they trusted the fact that the CPU frequency would increase, making their program faster without changing one single line. Today however, "the free lunch is over", as already argued by Herb Sutter in 2004. The frequency of CPUs tends to stagnate, while the number of cores in even the cheapest laptop has increased. For Sutter, the immediate consequence is: "applications will increasingly need to be concurrent if they want to fully exploit CPU throughput gains".Python is no exception to this paradigm change and this book introduces several ways to go parallel with Python using the standard library modules threading, multiprocessing and asyncio, and the third-party modules Parallel Python and Celery. This is quite an impressive list for such a short book, but I personally think that IPython.parallel and mpi4py (both quite popular in scientific computing) should have made it into this book.Parallel programming in Python is no trivial task because of the Global Interpreter Lock (GIL) which "prevents multiple native threads from executing Python bytecodes at once". While this issue is briefly mentioned in this book (in a section called "Taking Care of Python GIL"), I think that the author should have made it clearer which modules suffer from this limitation, and which don't. For example, the following statement (which can be found page 30) is far too vague: "Within the Python programming language, the use of CPU-bound threads may harm performance of the application due to GIL.""may harm performance"? Could you be more precise, please? The answer can be found in the module's official documentation: "CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing or concurrent.futures.ProcessPoolExecutor. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously."So the truth is that "may harm" should really read "does harm". To be fair, a much more clear-cut statement can be found page 94: "A way to solve this problem is delegating a blocking task to ThreadPoolExecutor (remember this works well if the processing is I/O bound; if it is CPU-bound, use ProcessPoolExecutor)".The book starts with two introductory chapters, in which the reader will find useful material. Chapter 1 presents a historical background, defines various forms of parallelization as well as the most common pitfalls of parallel programming (deadlock, starvation, race condition). Chapter 2 reviews a few parallelization strategies.In chapter 3, the author presents two simple problems which will be used in the remainder of the book to illustrate the use of the various modules I mentioned above. I find the idea of reusing the same two examples over and over again quite interesting, as it makes it easier to the reader to understand the differences and similarities between these modules. Besides, the two examples are simple, and so is their parallel implementation. Maybe these two problems are too simple, though. Indeed, both are embarrasingly parallel problems, for which very little communication is required (besides scattering the data at the beginning and gathering the results at the end). In such a favorable situation, the potential "parallel programming problems" listed in chapter 1 are simply swept under the rug, which is probably better for an introductory book on parallel programming.In Chapters 4, 5, 6, 7 and 8, these two problems are implemented using the following modules in turn: threading, multiprocessing, pp, celery and asyncio. Of course, the API of these modules is not (cannot be) described in detail in such a short book: only the most elementary classes and functions are described. So be ready to dive into the API documentations after reading this book! It should be mentioned that the chapters on celery and asyncio (new in Python 3.4) provide more details, and are very enjoyable. I do think that a closing comparison between all these modules is clearly missing.To conclude, I would recommend this book to those of you who are willing to go parallel in Python, but do not know where to start. This book usefully lists various options and provides the keys to each of those. Happy reading!

Parallel Programming with Python: Develop efficient parallel systems using the robust Python environment.

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Description

What you will learn

Product Details

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

About the author

FAQs

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access