Parallel list processing with execnet

This recipe presents a pattern for using execnet to process a list in parallel. It's a function pattern for mapping each element in the list to a new value, using execnet to do the mapping in parallel.

How to do it...

First, we need to decide exactly what we want to do. In this example, we'll just double integers, but we could do any pure computation. Following is the remote_double.py module, which will be executed by execnet. It receives a 2-tuple of (i, arg), assumes arg is a number, and sends back (i, arg*2). The need for i will be explained in the next section.

if __name__ == '__channelexec__':
  for (i, arg) in channel:
    channel.send((i, arg * 2))

To use this module to double every element in a list, we import the plists module (explained in the How it works... section) and call plists.map() with the remote_double module, and a list of integers to double.

>>> import plists, remote_double
>>> plists.map(remote_double, range(10))
...