IPython Interactive Computing and Visualization Cookbook

We assume that a Python distribution is installed with IPython and that we are now in an IPython notebook. We type the following command in a cell, and press Shift + Enter to evaluate it:

In [1]: print("Hello world!")
Hello world!

Screenshot of the IPython notebook

A notebook contains a linear succession of cells and output areas. A cell contains Python code, in one or multiple lines. The output of the code is shown in the corresponding output area.

Now, we do a simple arithmetic operation:

In [2]: 2+2
Out[2]: 4

The result of the operation is shown in the output area. Let's be more precise. The output area not only displays the text that is printed by any command in the cell, but it also displays a text representation of the last returned object. Here, the last returned object is the result of 2+2, that is, 4.

In the next cell, we can recover the value of the last returned object with the _ (underscore) special variable. In practice, it might be more convenient to assign objects to named variables such as in myresult = 2+2.

In [3]: _ * 3
Out[3]: 12

IPython not only accepts Python code, but also shell commands. These commands are defined by the operating system (mainly Windows, Linux, and Mac OS X). We first type ! in a cell before typing the shell command. Here, assuming a Linux or Mac OS X system, we get the list of all the notebooks in the current directory:

In [4]: !ls *.ipynb
notebook1.ipynb  ...

On Windows, you should replace ls with dir.

IPython comes with a library of magic commands. These commands are convenient shortcuts to common actions. They all start with % (the percent character). We can get the list of all magic commands with %lsmagic:

In [5]: %lsmagic
Out[5]: Available line magics:
%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cd  %clear  %cls  %colors  %config  %connect_info  %copy  %ddir  %debug  %dhist  %dirs  %doctest_mode  %echo  %ed  %edit  %env  %gui  %hist  %history  %install_default_config  %install_ext  %install_profiles  %killbgscripts  %ldir  %less  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %macro  %magic  %matplotlib  %mkdir  %more  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %ren  %rep  %rerun  %reset  %reset_selective  %rmdir  %run  %save  %sc  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%cmd  %%debug  %%file  %%html  %%javascript  %%latex  %%perl  %%powershell  %%prun  %%pypy  %%python  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Cell magics have a %% prefix; they concern entire code cells.

For example, the %%writefile cell magic lets us create a text file easily. This magic command accepts a filename as an argument. All the remaining lines in the cell are directly written to this text file. Here, we create a file test.txt and write Hello world! in it:

In [6]: %%writefile test.txt
        Hello world!
Writing test.txt
In [7]: # Let's check what this file contains.
        with open('test.txt', 'r') as f:
            print(f.read())
Hello world!

As we can see in the output of %lsmagic, there are many magic commands in IPython. We can find more information about any command by adding ? after it. For example, to get some help about the %run magic command, we type %run? as shown here:

In [9]: %run?
Type:        Magic function
Namespace:   IPython internal
...
Docstring:
Run the named file inside IPython as a program.
[full documentation of the magic command...]

We covered the basics of IPython and the notebook. Let's now turn to the rich display and interactive features of the notebook. Until now, we have only created code cells (containing code). IPython supports other types of cells. In the notebook toolbar, there is a drop-down menu to select the cell's type. The most common cell type after the code cell is the Markdown cell.

Markdown cells contain rich text formatted with Markdown, a popular plain text-formatting syntax. This format supports normal text, headers, bold, italics, hypertext links, images, mathematical equations in LaTeX (a typesetting system adapted to mathematics), code, HTML elements, and other features, as shown here:

### New paragraph 
This is *rich* **text** with [links](http://ipython.org),
equations:

$$\hat{f}(\xi) = \int_{-\infty}^{+\infty} f(x)\,
                 \mathrm{e}^{-i \xi x}$$

code with syntax highlighting: 
```python
print("Hello world!")
``` 
and images: 
![This is an image](http://ipython.org/_static/IPy_header.png)

Running a Markdown cell (by pressing Shift + Enter, for example) displays the output, as shown in the following screenshot:

Rich text formatting with Markdown in the IPython notebook

By combining code cells and Markdown cells, we can create a standalone interactive document that combines computations (code), text, and graphics.

IPython also comes with a sophisticated display system that lets us insert rich web elements in the notebook. Here, we show how to add HTML, SVG (Scalable Vector Graphics), and even YouTube videos in a notebook.

First, we need to import some classes:

In [11]: from IPython.display import HTML, SVG, YouTubeVideo

We create an HTML table dynamically with Python, and we display it in the notebook:

In [12]: HTML('''
         <table style="border: 2px solid black;">
         ''' + 
         ''.join(['<tr>' + 
                  ''.join(['<td>{row},{col}</td>'.format(
                                 row=row, col=col
                                 ) for col in range(5)]) +
                  '</tr>' for row in range(5)]) +
         '''
         </table>
         ''')

An HTML table in the notebook

Similarly, we can create SVG graphics dynamically:

In [13]: SVG('''<svg width="600" height="80">''' + 
         ''.join(['''<circle cx="{x}" cy="{y}" r="{r}"
                             fill="red"
                             stroke-width="2"
                             stroke="black">
                     </circle>'''.format(x=(30+3*i)*(10-i),
                                         y=30,
                                         r=3.*float(i)
                                         ) for i in range(10)]) + 
         '''</svg>''')

SVG in the notebook

Finally, we display a YouTube video by giving its identifier to YoutubeVideo:

In [14]: YouTubeVideo('j9YpkSX7NNM')

YouTube in the notebook

Now, we illustrate the latest interactive features in IPython 2.0+, namely JavaScript widgets. Here, we create a drop-down menu to select videos:

In [15]: from collections import OrderedDict
         from IPython.display import (display,
                                      clear_output,
                                      YouTubeVideo)
         from IPython.html.widgets import DropdownWidget
In [16]: # We create a DropdownWidget, with a dictionary
         # containing the keys (video name) and the values
         # (Youtube identifier) of every menu item.
         dw = DropdownWidget(values=OrderedDict([
                           ('SciPy 2012', 'iwVvqwLDsJo'),
                           ('PyCon 2012', '2G5YTlheCbw'),
                           ('SciPy 2013', 'j9YpkSX7NNM')]
                                                )
                             )
         
         # We create a callback function that displays the 
         # requested Youtube video.
         def on_value_change(name, val):
             clear_output()
             display(YouTubeVideo(val))
         
         # Every time the user selects an item, the 
         # function `on_value_change` is called, and the 
         # `val` argument contains the value of the selected 
         # item.
         dw.on_trait_change(on_value_change, 'value')
         
         # We choose a default value.
         dw.value = dw.values['SciPy 2013']
         
         # Finally, we display the widget.
         display(dw)

An interactive widget in the notebook

The very first step is to import the scientific packages we will be using in this recipe, namely NumPy, pandas, and matplotlib. We also instruct matplotlib to render the figures as inline images in the notebook:

In [1]: import numpy as np
        import pandas as pd
        import matplotlib.pyplot as plt
        %matplotlib inline

Now, we create a new Python variable called url that contains the address to a CSV (Comma-separated values) data file. This standard text-based file format is used to store tabular data:

In [2]: url = "http://donnees.ville.montreal.qc.ca/storage/f/2014-01-20T20%3A48%3A50.296Z/2013.csv"

pandas defines a read_csv() function that can read any CSV file. Here, we pass the URL to the file. pandas will automatically download and parse the file, and return a DataFrame object. We need to specify a few options to make sure that the dates are parsed correctly:

In [3]: df = pd.read_csv(url, index_col='Date',
                         parse_dates=True, dayfirst=True)

The df variable contains a DataFrame object, a specific pandas data structure that contains 2D tabular data. The head(n) method displays the first n rows of this table. In the notebook, pandas displays a DataFrame object in an HTML table, as shown in the following screenshot:

In [4]: df.head(2)

First rows of the DataFrame

Here, every row contains the number of bicycles on every track of the city, for every day of the year.

We can get some summary statistics of the table with the describe() method:

In [5]: df.describe()

Summary statistics of the DataFrame

Let's display some figures. We will plot the daily attendance of two tracks. First, we select the two columns, Berri1 and PierDup. Then, we call the plot() method:

In [6]: df[['Berri1', 'PierDup']].plot()

Now, we move to a slightly more advanced analysis. We will look at the attendance of all tracks as a function of the weekday. We can get the weekday easily with pandas: the index attribute of the DataFrame object contains the dates of all rows in the table. This index has a few date-related attributes, including weekday:

In [7]: df.index.weekday
Out[7]: array([1, 2, 3, 4, 5, 6, 0, 1, 2, ..., 0, 1, 2])

However, we would like to have names (Monday, Tuesday, and so on) instead of numbers between 0 and 6. This can be done easily. First, we create a days array with all the weekday names. Then, we index it by df.index.weekday. This operation replaces every integer in the index by the corresponding name in days. The first element, Monday, has the index 0, so every 0 in df.index.weekday is replaced by Monday and so on. We assign this new index to a new column, Weekday, in DataFrame:

In [8]: days = np.array(['Monday', 'Tuesday', 'Wednesday', 
                         'Thursday', 'Friday', 'Saturday', 
                         'Sunday'])
        df['Weekday'] = days[df.index.weekday]

To get the attendance as a function of the weekday, we need to group the table elements by the weekday. The groupby() method lets us do just that. Once grouped, we can sum all the rows in every group:

In [9]: df_week = df.groupby('Weekday').sum()
In [10]: df_week

Grouped data with pandas

We can now display this information in a figure. We first need to reorder the table by the weekday using ix (indexing operation). Then, we plot the table, specifying the line width:

In [11]: df_week.ix[days].plot(lw=3)
         plt.ylim(0);  # Set the bottom axis to 0.

Finally, let's illustrate the new interactive capabilities of the notebook in IPython 2.0. We will plot a smoothed version of the track attendance as a function of time (rolling mean). The idea is to compute the mean value in the neighborhood of any day. The larger the neighborhood, the smoother the curve. We will create an interactive slider in the notebook to vary this parameter in real time in the plot. All we have to do is add the @interact decorator above our plotting function:

In [12]: from IPython.html.widgets import interact
         @interact
         def plot(n=(1, 30)):
             pd.rolling_mean(df['Berri1'], n).dropna().plot()
             plt.ylim(0, 8000)
             plt.show()

Interactive widget in the notebook

Let's import the built-in random Python module and NumPy:

In [1]: import random
        import numpy as np

We use the %precision magic (defined in IPython) to show only three decimals in the Python output. This is just to reduce the number of digits in the output's text.

In [2]: %precision 3
Out[2]: u'%.3f'

We generate two Python lists, x and y, each one containing 1 million random numbers between 0 and 1:

In [3]: n = 1000000
        x = [random.random() for _ in range(n)]
        y = [random.random() for _ in range(n)]
In [4]: x[:3], y[:3]
Out[4]: ([0.996, 0.846, 0.202], [0.352, 0.435, 0.531])

Let's compute the element-wise sum of all these numbers: the first element of x plus the first element of y, and so on. We use a for loop in a list comprehension:

In [5]: z = [x[i] + y[i] for i in range(n)]
        z[:3]
Out[5]: [1.349, 1.282, 0.733]

How long does this computation take? IPython defines a handy %timeit magic command to quickly evaluate the time taken by a single statement:

In [6]: %timeit [x[i] + y[i] for i in range(n)]
1 loops, best of 3: 273 ms per loop

Now, we will perform the same operation with NumPy. NumPy works on multidimensional arrays, so we need to convert our lists to arrays. The np.array() function does just that:

In [7]: xa = np.array(x)
        ya = np.array(y)
In [8]: xa[:3]
Out[8]: array([ 0.996,  0.846,  0.202])

The xa and ya arrays contain the exact same numbers that our original lists, x and y, contained. Those lists were instances of the list built-in class, while our arrays are instances of the ndarray NumPy class. These types are implemented very differently in Python and NumPy. In this example, we will see that using arrays instead of lists leads to drastic performance improvements.

Now, to compute the element-wise sum of these arrays, we don't need to do a for loop anymore. In NumPy, adding two arrays means adding the elements of the arrays component-by-component. This is the standard mathematical notation in linear algebra (operations on vectors and matrices):

In [9]: za = xa + ya
        za[:3]
Out[9]: array([ 1.349,  1.282,  0.733])

We see that the z list and the za array contain the same elements (the sum of the numbers in x and y).

Let's compare the performance of this NumPy operation with the native Python loop:

In [10]: %timeit xa + ya
100 loops, best of 3: 10.7 ms per loop

We observe that this operation is more than one order of magnitude faster in NumPy than in pure Python!

Now, we will compute something else: the sum of all elements in x or xa. Although this is not an element-wise operation, NumPy is still highly efficient here. The pure Python version uses the built-in sum() function on an iterable. The NumPy version uses the np.sum() function on a NumPy array:

In [11]: %timeit sum(x)  # pure Python
         %timeit np.sum(xa)  # NumPy
100 loops, best of 3: 17.1 ms per loop
1000 loops, best of 3: 2.01 ms per loop

We also observe an impressive speedup here also.

Finally, let's perform one last operation: computing the arithmetic distance between any pair of numbers in our two lists (we only consider the first 1000 elements to keep computing times reasonable). First, we implement this in pure Python with two nested for loops:

In [12]: d = [abs(x[i] - y[j]) 
              for i in range(1000) for j in range(1000)]
In [13]: d[:3]
Out[13]: [0.230, 0.037, 0.549]

Now, we use a NumPy implementation, bringing out two slightly more advanced notions. First, we consider a two-dimensional array (or matrix). This is how we deal with the two indices, i and j. Second, we use broadcasting to perform an operation between a 2D array and 1D array. We will give more details in the How it works... section.

In [14]: da = np.abs(xa[:1000,None] - ya[:1000])
In [15]: da
Out[15]: array([[ 0.23 ,  0.037,  ...,  0.542,  0.323,  0.473],
                 ...,
                [ 0.511,  0.319,  ...,  0.261,  0.042,  0.192]])
In [16]: %timeit [abs(x[i] - y[j]) 
                  for i in range(1000) for j in range(1000)]
         %timeit np.abs(xa[:1000,None] - ya[:1000])
1 loops, best of 3: 292 ms per loop
100 loops, best of 3: 18.4 ms per loop

Here again, we observe significant speedups.

Let's import a few functions from the IPython magic system:

In [1]: from IPython.core.magic import (register_line_magic, 
                                        register_cell_magic)

Defining a new line magic is particularly simple. First, we create a function that accepts the contents of the line (except the initial %-prefixed name). The name of this function is the name of the magic. Then, we decorate this function with @register_line_magic:

In [2]: @register_line_magic
        def hello(line):
            if line == 'french':
                print("Salut tout le monde!")
            else:
                print("Hello world!")
In [3]: %hello
Hello world!
In [4]: %hello french
Salut tout le monde!

Let's create a slightly more useful %%csv cell magic that parses a CSV string and returns a pandas DataFrame object. This time, the arguments of the function are the characters that follow %%csv in the first line and the contents of the cell (from the cell's second line to the last).

In [5]: import pandas as pd
        #from StringIO import StringIO  # Python 2
        from io import StringIO  # Python 3

        @register_cell_magic
        def csv(line, cell):
            # We create a string buffer containing the
            # contents of the cell.
            sio = StringIO(cell)
            # We use pandas' read_csv function to parse
            # the CSV string.
            return pd.read_csv(sio)
In [6]: %%csv
        col1,col2,col3
        0,1,2
        3,4,5
        7,8,9
Out[6]:
    col1  col2  col3
0     0     1     2
1     3     4     5
2     7     8     9

We can access the returned object with _.

In [7]: df = _
        df.describe()
Out[7]:
           col1      col2      col3
count  3.000000  3.000000  3.000000
mean   3.333333  4.333333  5.333333
...
min    0.000000  1.000000  2.000000
max    7.000000  8.000000  9.000000

The method we described is useful in an interactive session. If we want to use the same magic in multiple notebooks or if we want to distribute it, then we need to create an IPython extension that implements our custom magic command. The first step is to create a Python script (csvmagic.py here) that implements the magic. We also need to define a special function load_ipython_extension(ipython):

In [8]: %%writefile csvmagic.py
        import pandas as pd
        #from StringIO import StringIO  # Python 2
        from io import StringIO  # Python 3
        def csv(line, cell):
            sio = StringIO(cell)
            return pd.read_csv(sio)

        def load_ipython_extension(ipython):
            """This function is called when the extension 
            is loaded. It accepts an 
            IPython InteractiveShell instance.
            We can register the magic with the
            `register_magic_function` method."""
            ipython.register_magic_function(csv, 'cell')
Overwriting csvmagic.py

Once the extension module is created, we need to import it into the IPython session. We can do this with the %load_ext magic command. Here, loading our extension immediately registers our %%csv magic function in the interactive shell:

In [9]: %load_ext csvmagic
In [10]: %%csv
         col1,col2,col3
         0,1,2
         3,4,5
         7,8,9
Out[10]:
   col1  col2  col3
0     0     1     2
1     3     4     5
2     7     8     9

We create an IPython extension in a random_magics.py file. Let's start by importing a few objects.

from IPython.utils.traitlets import Int, Float, Unicode, Bool
from IPython.core.magic import (Magics, magics_class, line_magic)
import numpy as np

We create a RandomMagics class deriving from Magics. This class contains a few configurable parameters:

@magics_class
class RandomMagics(Magics):
    text = Unicode(u'{n}', config=True)
    max = Int(1000, config=True)
    seed = Int(0, config=True)

We need to call the parent's constructor. Then, we initialize a random number generator with a seed:

    def __init__(self, shell):
        super(RandomMagics, self).__init__(shell)
        self._rng = np.random.RandomState(self.seed or None)

We create a %random line magic that displays a random number:

    @line_magic
    def random(self, line):
        return self.text.format(n=self._rng.randint(self.max))

Finally, we register that magic when the extension is loaded:

def load_ipython_extension(ipython):
    ipython.register_magics(RandomMagics)

Let's test our extension in the notebook:

In [1]: %load_ext random_magics
In [2]: %random
Out[2]: '635'
In [3]: %random
Out[3]: '47'

Our magic command has a few configurable parameters. These variables are meant to be configured by the user in the IPython configuration file or in the console when starting IPython. To configure these variables in the terminal, we can type the following command in a system shell:

ipython --RandomMagics.text='Your number is {n}.' --RandomMagics.max=10 --RandomMagics.seed=1

In this session, we get the following behavior:

In [1]: %load_ext random_magics
In [2]: %random
Out[2]: u'Your number is 5.'
In [3]: %random
Out[3]: u'Your number is 8.'

To configure the variables in the IPython configuration file, we have to open the ~/.ipython/profile_default/ipython_config.py file and add the following line:

c.RandomMagics.text = 'random {n}'

After launching IPython, we get the following behavior:

In [4]: %random
Out[4]: 'random 652'

First, we create a plotkernel.py file. This file will contain the implementation of our custom kernel. Let's import a few modules:

from IPython.kernel.zmq.kernelbase import Kernel
import numpy as np
import matplotlib.pyplot as plt
from io import BytesIO
import urllib, base64

We write a function that returns a base64-encoded PNG representation of a matplotlib figure:

def _to_png(fig):
    """Return a base64-encoded PNG from a 
    matplotlib figure."""
    imgdata = BytesIO()
    fig.savefig(imgdata, format='png')
    imgdata.seek(0)
    return urllib.parse.quote(
        base64.b64encode(imgdata.getvalue()))

Now, we write a function that parses a code string, which has the form y = f(x), and returns a NumPy function. Here, f is an arbitrary Python expression that can use NumPy functions:

_numpy_namespace = {n: getattr(np, n) 
                    for n in dir(np)}
def _parse_function(code):
    """Return a NumPy function from a string 'y=f(x)'."""
    return lambda x: eval(code.split('=')[1].strip(),
                          _numpy_namespace, {'x': x})

For our new wrapper kernel, we create a class that derives from Kernel. There are a few metadata fields we need to provide:

class PlotKernel(Kernel):
    implementation = 'Plot'
    implementation_version = '1.0'
    language = 'python'  # will be used for
                         # syntax highlighting
    language_version = ''
    banner = "Simple plotting"

In this class, we implement a do_execute() method that takes code as input and sends responses to the client:

def do_execute(self, code, silent,
               store_history=True,
               user_expressions=None,
               allow_stdin=False):

    # We create the plot with matplotlib.
    fig = plt.figure(figsize=(6,4), dpi=100)
    x = np.linspace(-5., 5., 200)
    functions = code.split('\n')
    for fun in functions:
        f = _parse_function(fun)
        y = f(x)
        plt.plot(x, y)
    plt.xlim(-5, 5)

    # We create a PNG out of this plot.
    png = _to_png(fig)

    if not silent:
        # We send the standard output to the client.
        self.send_response(self.iopub_socket,
            'stream', {
                'name': 'stdout', 
                'data': 'Plotting {n} function(s)'. \
                            format(n=len(functions))})

        # We prepare the response with our rich data
        # (the plot).
        content = {
            'source': 'kernel',

            # This dictionary may contain different
            # MIME representations of the output.
            'data': {
                'image/png': png
            },

            # We can specify the image size
            # in the metadata field.
            'metadata' : {
                  'image/png' : {
                    'width': 600,
                    'height': 400
                  }
                }
        }        

        # We send the display_data message with the
        # contents.
        self.send_response(self.iopub_socket,
            'display_data', content)

    # We return the execution results.
    return {'status': 'ok',
            'execution_count': self.execution_count,
            'payload': [],
            'user_expressions': {},
           }

Finally, we add the following lines at the end of the file:

if __name__ == '__main__':
    from IPython.kernel.zmq.kernelapp import IPKernelApp
    IPKernelApp.launch_instance(kernel_class=PlotKernel)

Our kernel is ready! The next step is to indicate to IPython that this new kernel is available. To do this, we need to create a kernel spec kernel.json file and put it in ~/.ipython/kernels/plot/. This file contains the following lines:

{
 "argv": ["python", "-m",
          "plotkernel", "-f",
          "{connection_file}"],
 "display_name": "Plot",
 "language": "python"
}

The plotkernel.py file needs to be importable by Python. For example, we could simply put it in the current directory.

In IPython 3.0, we can launch a notebook with this kernel from the IPython notebook dashboard. There is a drop-down menu at the top right of the notebook interface that contains the list of available kernels. Select the Plot kernel to use it.

Finally, in a new notebook backed by our custom plot kernel, we can simply write the mathematical equation, y = f(x). The corresponding graph appears in the output area. Here is an example:

Example of our custom plot wrapper kernel