About this book

You already use Python as a scripting language, but did you know it is also increasingly used for scientific computing and data analysis? Interactive programming is essential in such exploratory tasks and IPython is the perfect tool for that. Once you’ve learnt it, you won’t be able to live without it.

"Learning IPython for Interactive Computing and Data Visualization" is a practical, hands-on, example-driven tutorial to considerably improve your productivity during interactive Python sessions, and shows you how to effectively use IPython for interactive computing and data analysis.

This book covers all aspects of IPython, from the highly powerful interactive Python console to the numerical and visualization features that are commonly associated with IPython.

You will learn how IPython lets you perform efficient vectorized computations, through examples covering numerical simulations with NumPy, data analysis with Pandas, and visualization with Matplotlib. You will also discover how IPython can be conveniently used to optimize your code using parallel computing and dynamic compilation in C with Cython.

"Learning IPython for Interactive Computing and Data Visualization" will allow you to optimize your productivity in interactive Python sessions.

Publication date:
April 2013
Publisher
Packt
Pages
138
ISBN
9781782169932

 

Chapter 1. Getting Started with IPython

In this chapter, we will first go through the IPython installation process and give an overview of the possibilities offered by IPython. IPython brings a highly improved Python console and the Notebook. In addition, it is an essential tool for interactive computing when it is combined with third-party specialized packages, such as NumPy and Matplotlib. These packages bring high-performance computing and interactive visualization facilities to the Python universe, with IPython being its cornerstone. At the end of this chapter, you will have IPython installed and the required packages on your computer, and you will have been through a short, hands-on overview of the most important features of IPython that we will detail in the subsequent chapters, such as:

  • Running the IPython console

  • Using IPython as a system shell

  • Using the history

  • Tab completion

  • Executing a script with the %run command

  • Quick benchmarking with the %timeit command

  • Quick debugging with the %pdb command

  • Interactive computing with Pylab

  • Using the IPython Notebook

  • Customizing IPython

 

Installing IPython and the recommended packages


In this section, we will see how you can install IPython and the other packages that we will be using in this book. For the most up-to-date information about the IPython installation, you should check the official website of IPython (http://ipython.org).

Prerequisites for IPython

First things first, what do you need to have on your computer before installing IPython? The good news is that IPython, and more generally all Python packages, can run, in principle, on most platforms (that is, Linux, Apple OS X, and Microsoft Windows). You also need to have a valid Python distribution installed on your system before installing and running IPython. The latest stable version of IPython at the time of writing is 0.13.1, and it officially requires Python 2.6, 2.7, 3.1, or 3.2.

Tip

Python 2.x and 3.x

The 3.x branch of Python is not backward compatible with the 2.x branch, which explains why the 2.7 version is still maintained. Even if most external Python packages used in this book are compatible with Python 3.x, some packages are still not compatible with this branch. At this time, the choice between Python 2.x and Python 3.x for a new project is typically dictated by the Python 3 support of the required external Python packages. The setups of the targeted users is also an important point to consider. In this book, we will use Python 2.7 and try to minimize the incompatibilities with Python 3.x. This issue is beyond the scope of this book, and we encourage you to search for information about how to write code for Python 2.x that is as compatible with Python 3.x as possible. This official web page is a good starting point:

http://wiki.python.org/moin/Python2orPython3

We will use Python 2.7 in this book. The 2.6 version is no longer maintained and, if you choose to stick with the 2.x branch, you should only use Python 2.7 as far as possible.

We will use other Python packages in this book that are typically used with IPython. These packages are mainly NumPy, SciPy, and Matplotlib, but there are additional packages we will use in some examples. Details about how to install them are provided in the next section Installing an all-in-one distribution.

There are several ways of installing IPython and the recommended packages. From the easiest to the hardest, you can do either of the following:

  • Install a standalone, all-in-one Python distribution with a large variety of built-in Python packages

  • Install separately only the packages you need

In the latter case, you can use binary installers or install the packages directly from the source code.

Installing an all-in-one distribution

This solution is by far the easiest. You can download a single binary installer that comes with a full Python distribution and a lot of widely used external packages, including IPython. Popular distributions include:

All these distributions support Linux, OS X, and Windows, except Python(x,y) which only supports Windows. They all offer a free edition (and possibly a commercial edition) and they all contain IPython. ActivePython and EPD also ship with their own packaging systems; this makes it easy to install additional packages. These distributions contain most of the external packages we will be using in this book.

Installing the packages one by one

Sometimes you may prefer to install only the packages you need instead of installing a large all-in-one package. Fortunately, this should be straightforward on most recent systems. Binary installers are indeed available for Windows, OS X, and most common Linux distributions. Otherwise, there is always the possibility to install the packages from the source, which is generally easier than it sounds.

Packages websites

Here is a list of Python packages that we will mention in this book, along with their respective websites where you can find the most up-to-date information:

Tip

PyQt or PySide?

Qt is a cross-platform application framework widely used for software with GUI. It has a complex history; originally developed by Trolltech, it was then acquired by Nokia and now owned by Digia. Both commercial and open source licenses exist. PyQt is a Qt wrapper in Python developed by Riverbank Computing. The open source version of PyQt is GPL licensed, which prevents using it in commercial products. Therefore, Nokia decided to create its own LGPL-licensed package called PySide. It is now maintained by the Qt Project. Today, both packages coexist and have an extremely similar API so that it is possible to write Qt graphical applications in Python that support both libraries.

These websites propose to download binary installers for various systems as well as the source code for manual compilation and installation.

There is also an online repository of Python packages called the Python Package Index (PyPI) available at http://pypi.python.org. It contains tarballs, and sometimes Windows installers, for most existing Python packages.

Getting binary installers

You may find a binary installer for your system on the official website of the packages you are interested in. If official binary installers are not available, unofficial ones may have been created by the community. We will give some advice here about where binary installers can be found on the different operating systems.

Windows

Official Windows installers may be found on the package websites or on PyPI for some packages. Unofficial Windows installers for hundreds of Python packages (including IPython and all the packages used in this book) can be found on the personal webpage of Christoph Gohlke at http://www.lfd.uci.edu/~gohlke/pythonlibs/. These files are provided without warranty of any kind. However, they are generally quite stable, and this makes it extremely easy to install almost any Python package on Windows. There are versions of all packages for Python 2.x and 3.x and for 32-bit and 64-bit Python distributions.

OS X

Official OS X installers can be found on the websites of some packages, and unofficial installers can be found on the MacPorts project (http://www.macports.org) and Homebrew (http://mxcl.github.com/homebrew/).

Linux

Most Linux distributions (including Ubuntu) ship with a packaging system that may contain the Python version you need along with most Python packages we will be using here. For example, to install IPython on Ubuntu, type the following command in a shell:

$ sudo apt-get install ipython-notebook

On Fedora 18 and newer related distributions, type the following command:

$ sudo yum install python-ipython-notebook

The relevant binary package names are sometimes prefixed with python- (for example, python-numpy or python-matplotlib). Also, PyQt4's package name is python-qt4, PyOpenGL's package name is python-opengl, PIL's package name is python-imaging, and so on.

Table of binary packages

We have shown here a table with the availability (at the time of writing) of binary installers for the packages we will be using in this book in the different Python distributions and operating systems. All these installers are available for Python 2.7. In the following table, "(W)" means Windows and "CG:" means Christoph Gohlke's webpage:

Package

EPD 7.3

Anaconda 1.2.1

Python(x,y) 2.7.3

ActivePython 2.7.2

Windows installer

Ubuntu installer

OSX installer (MacPorts)

NetworkX

1.6

1.7

1.7

1.6

CG: 1.7

1.7

1.7

Pandas

0.9.1

0.9.0

0.9.1

0.7.3

CG: 0.10.0, PyPI: 0.10.0

0.8.0

0.10.0

NumPy

1.6.1

1.6.2 (W)

1.6.2

1.6.2

CG: 1.6.2, PyPI: 1.6.2 (32 bits)

1.6.2

1.6.2

SciPy

0.10.1

0.11.0

0.11.0

0.10.1

CG: 0.11.0

0.10.1

0.11.0

PIL

1.1.7

1.1.7

1.1.7

1.1.7

CG: 1.1.7

1.1.7

N/A

Matplotlib

1.1.0

1.2.0

1.1.1

1.1.0

CG: 1.2.0

1.1.1

1.2.0

Basemap

1.0.2

N/A

1.0.2 (optional)

1.0 beta

1.0.5

1.0.5

1.0.5

PyOpenGL

3.0.1

N/A

3.0.2

3.0.2

CG: 3.0.2, PyPI: 3.0.2

3.0.1

3.0.2

PySide

1.1.1

1.1.2

N/A (PyQt 4.9.5)

N/A (PyQt 4.8.3)

CG: 1.1.2

1.1.1

1.1.2

Cython

0.16

0.17.1

0.17.2

0.16

CG: 0.17.3

0.16

0.17.3

Numba

N/A

0.3.2

N/A

N/A

CG: 0.3.2

N/A

N/A

Using the Python packaging system

When binary packages are not available, the universal way of installing a Python package is to install it directly from its source code. The Python packaging system is meant to simplify this step so as to handle dependency management, uninstallation, and package discovery. However, the packaging system has been chaotic for years.

Distutils, the native Python packaging system, has long been criticized for being inefficient and bringing too many problems. Its successor Distutils2 is not finished at the time of writing. Setuptools is an alternative system and offers the easy_install command-line tool that allows searching (on PyPI) and installing new Python packages with a single command line. Installing a new package is as simple as typing in a shell:

$ easy_install ipython

Setuptools has also been criticized and is now being replaced by Distribute. The easy_install tool is also being replaced by pip, a more powerful tool for searching, installing, and uninstalling Python packages.

For now, we recommend that you use Distribute and pip. Both can be installed either from the source tarballs or with easy_install (which requires that you install Setuptools beforehand). More details about how to install these tools can be found on The Hitchhiker's Guide to Packaging (http://guide.python-distribute.org/). To install a new package with pip, type the following command in a shell:

$ pip install ipython
Optional dependencies for IPython

IPython has several dependencies:

  • pyreadline: This dependency provides line-editing features

  • pyzmq: This dependency is needed for IPython's parallel computing features, such as Qt console and Notebook

  • pygments: This dependency highlights syntax in the Qt console

  • tornado: This dependency is required by the web-based Notebook

They are all automatically installed when you install IPython from a binary package, but that is not the case when you install IPython from the source code. On Windows, pyreadline must be installed using either a binary installer available on PyPI or on Christoph Gohlke's webpage, or with easy_install or pip.

On OS X, you should also install readline with easy_install or pip.

The other dependencies can automatically be installed with the following command:

$ easy_install ipython[zmq,qtconsole,notebook]

Installing the development versions

The most experienced users may want to use the very latest development versions of some libraries. Details can be found on the websites of the respective libraries. For example, to install the development version of IPython, we can type the following command (the version control system Git needs to be installed):

$ git clone https://github.com/ipython/ipython.git
$ cd ipython
$ python setup.py install

To be able to update IPython easily as it changes on the development branch (by using git pull), we can just replace the last line with the following command (the Distribute library needs to be installed):

$ python setupegg.py develop

Tip

Getting help for IPython

The official IPython documentation webpage at http://ipython.org/documentation.html is the place to go to get some help. It contains links to the online manual and to unofficial tutorials and articles created by the community. The StackOverflow website at http://stackoverflow.com/questions/tagged/ipython is also a great place to request help for IPython. Finally, anyone can subscribe to the IPython users' mailing list http://mail.scipy.org/mailman/listinfo/ipython-user.

 

Ten IPython essentials


In this section, we will take a quick tour of IPython by introducing 10 essential features of this powerful tool. Although brief, this hands-on visit will cover a wide range of IPython functionality that will be explored in more detail in the next chapters.

Running the IPython console

If IPython has been installed correctly, you should be able to run it from a system shell with the ipython command. You can use this prompt like a regular Python interpreter as shown in the following screenshot:

The IPython console

Tip

Command-line shell on Windows

If you are on Windows and using the old cmd.exe shell , you should be aware that this tool is extremely limited. You could instead use a more powerful interpreter, such as Microsoft PowerShell, which is integrated by default in Windows 7 and 8. The simple fact that most common filesystem-related commands (namely, pwd, cd, ls, cp, ps, and so on) have the same name as in Unix should be a sufficient reason to switch.

Of course, IPython offers much more than that. For example, IPython ships with tens of little commands that considerably improve productivity. We will see a lot of them in this book, starting with this section.

Some of these commands help you get information about any Python function or object. For instance, have you ever had a doubt about how to use the super function to access parent methods in a derived class? Just type super? (a shortcut for the command %pinfo super) and you will find all the information regarding the super function. Appending ? or ?? to any command or variable gives you all the information you need about it, as shown here:

In [1]: super?
Typical use to call a cooperative superclass method:
class C(B):
    def meth(self, arg):
        super(C, self).meth(arg)

Using IPython as a system shell

You can use the IPython command-line interface as an extended system shell. You can navigate throughout your filesystem and execute any system command. For instance, the standard Unix commands pwd, ls, and cd are available in IPython and work on Windows too, as shown in the following example:

In [1]: pwd
Out[1]: u'C:\\'
In [2]: cd windows
C:\windows

These commands are particular magic commands that are central in the IPython shell. There are dozens of magic commands and we will use a lot of them throughout this book. You can get a list of all magic commands with the %lsmagic command.

Tip

Using the IPython magic commands

Magic commands actually come with a % prefix, but the automagic system, enabled by default, allows you to conveniently omit this prefix. Using the prefix is always possible, particularly when the unprefixed command is shadowed by a Python variable with the same name. The %automagic command toggles the automagic system. In this book, we will generally use the % prefix to refer to magic commands, but keep in mind that you can omit it most of the time, if you prefer.

Using the history

Like the standard Python console, IPython offers a command history. However, unlike in Python's console, the IPython history spans your previous interactive sessions. In addition to this, several key strokes and commands allow you to reduce repetitive typing.

In an IPython console prompt, use the up and down arrow keys to go through your whole input history. If you start typing before pressing the arrow keys, only the commands that match what you have typed so far will be shown.

In any interactive session, your input and output history is kept in the In and Out variables and is indexed by a prompt number. The _, __, ___ and _i, _ii, _iii variables contain the last three output and input objects, respectively. The _n and _in variables return the nth output and input history. For instance, let's type the following command:

In [4]: a = 12
In [5]: a ** 2
Out[5]: 144
In [6]: print("The result is {0:d}.".format(_))
The result is 144.

In this example, we display the output, that is, 144 of prompt 5 on line 6.

Tab completion

Tab completion is incredibly useful and you will find yourself using it all the time. Whenever you start typing any command, variable name, or function, press the Tab key to let IPython either automatically complete what you are typing if there is no ambiguity, or show you the list of possible commands or names that match what you have typed so far. It also works for directories and file paths, just like in the system shell.

It is also particularly useful for dynamic object introspection. Type any Python object name followed by a point and then press the Tab key; IPython will show you the list of existing attributes and methods, as shown in the following example:

In [1]: import os
In [2]: os.path.split<TAB>
os.path.split os.path.splitdrive os.path.splitext os.path.splitunc

In the second line, as shown in the previous code, we press the Tab key after having typed os.path.split. IPython then displays all the possible commands.

Tip

Tab Completion and Private Variables

Tab completion shows you all the attributes and methods of an object, except those that begin with an underscore (_). The reason is that it is a standard convention in Python programming to prefix private variables with an underscore. To force IPython to show all private attributes and methods, type myobject._ before pressing the Tab key. Nothing is really private or hidden in Python. It is part of a general Python philosophy, as expressed by the famous saying, "We are all consenting adults here."

Executing a script with the %run command

Although essential, the interactive console becomes limited when running sequences of multiple commands. Writing multiple commands in a Python script with the .py file extension (by convention) is quite common. A Python script can be executed from within the IPython console with the %run magic command followed by the script filename. The script is executed in a fresh, new Python namespace unless the -i option has been used, in which case the current interactive Python namespace is used for the execution. In all cases, all variables defined in the script become available in the console at the end of script execution.

Let's write the following Python script in a file called script.py:

print("Running script.")
x = 12
print("'x' is now equal to {0:d}.".format(x))

Now, assuming we are in the directory where this file is located, we can execute it in IPython by entering the following command:

In [1]: %run script.py
Running script.
'x' is now equal to 12.
In [2]: x
Out[2]: 12

When running the script, the standard output of the console displays any print statement. At the end of execution, the x variable defined in the script is then included in the interactive namespace, which is quite convenient.

Quick benchmarking with the %timeit command

You can do quick benchmarks in an interactive session with the %timeit magic command. It lets you estimate how much time the execution of a single command takes. The same command is executed multiple times within a loop, and this loop itself is repeated several times by default. The individual execution time of the command is then automatically estimated with an average. The -n option controls the number of executions in a loop, whereas the -r option controls the number of executed loops. For example, let's type the following command:

In[1]: %timeit [x*x for x in range(100000)]
10 loops, best of 3: 26.1 ms per loop

Here, it took about 26 milliseconds to compute the squares of all integers up to 100000.

Quick debugging with the %debug command

IPython ships with a powerful command-line debugger. Whenever an exception is raised in the console, use the %debug magic command to launch the debugger at the exception point. You then have access to all the local variables and to the full stack traceback in postmortem mode. Navigate up and down through the stack with the u and d commands and exit the debugger with the q command. See the list of all the available commands in the debugger by entering the ? command.

You can use the %pdb magic command to activate the automatic execution of the IPython debugger as soon as an exception is raised.

Interactive computing with Pylab

The %pylab magic command enables the scientific computing capabilities of the NumPy and matplotlib packages, namely efficient operations on vectors and matrices and plotting and interactive visualization features. It becomes possible to perform interactive computations in the console and plot graphs dynamically. For example, let's enter the following command:

In [1]: %pylab
Welcome to pylab, a matplotlib-based Python environment [backend: TkAgg].
For more information, type 'help(pylab)'.
In [2]: x = linspace(-10., 10., 1000)
In [3]: plot(x, sin(x))

In this example, we first define a vector of 1000 values linearly spaced between -10 and 10. Then we plot the graph (x, sin(x)). A window with a plot appears as shown in the following screenshot, and the console is not blocked while this window is opened. This allows us to interactively modify the plot while it is open.

A Matplotlib figure

Using the IPython Notebook

The Notebook brings the functionality of IPython into the browser for multiline text-editing features, interactive session reproducibility, and so on. It is a modern and powerful way of using Python in an interactive and reproducible way.

To use the Notebook, call the ipython notebook command in a shell (make sure you have installed the required dependencies described in the Installation section). This will launch a local web server on the default port 8888. Go to http://127.0.0.1:8888/ in a browser and create a new Notebook.

You can write one or several lines of code in the input cells. Here are some of the most useful keyboard shortcuts:

  • Press the Enter key to create a new line in the cell and not execute the cell

  • Press Shift + Enter to execute the cell and go to the next cell

  • Press Alt + Enter to execute the cell and append a new empty cell right after it

  • Press Ctrl + Enter for quick instant experiments when you do not want to save the output

  • Press Ctrl + M and then the H key to display the list of all the keyboard shortcuts

We will explore the features of the Notebook more thoroughly in the next chapter.

Customizing IPython

You can save your user preferences in a Python file; this file is called an IPython profile. To create a default profile, type ipython profile create in a shell. This will create a folder named profile_default in the ~/.ipython or ~/.config/ipython directory. The file ipython_config.py in this folder contains preferences about IPython. You can create different profiles with different names using ipython profile create profilename, and then launch IPython with ipython --profile=profilename to use that profile.

The ~ directory is your home directory, for example, something like /home/yourname on Unix, or C:\Users\yourname or C:\Documents and Settings\yourname on Windows.

 

Summary


In this chapter, we have detailed the various ways with which you can install IPython and the recommended external Python packages. The most straightforward way is to install a standalone Python distribution with all packages built in, such as Enthought Python Distribution or Canopy, Anaconda, Python(x,y), or ActivePython, among others. The other solution is to install the different packages manually, either with binary installers available for most recent platforms, or by using the Python packaging system, which should be straightforward in most cases.

We have also gone through 10 of the most interesting features offered by IPython. They essentially concern the Python and shell interactive features, including the integrated debugger and profiler, and the interactive computing and visualization features brought by the NumPy and Matplotlib packages. In the following chapter, we will detail the interactive shell and Python console as well as the Notebook.

About the Author

  • Cyrille Rossant

    Cyrille Rossant, PhD, is a neuroscience researcher and software engineer at University College London. He is a graduate of École Normale Supérieure, Paris, where he studied mathematics and computer science. He has also worked at Princeton University and Collège de France. While working on data science and software engineering projects, he gained experience in numerical computing, parallel computing, and high-performance data visualization.

    He is the author of Learning IPython for Interactive Computing and Data Visualization, Second Edition, Packt Publishing.

    Browse publications by this author

Latest Reviews

(7 reviews total)
Excellent
Great technical resource for self learners
Writing interactive code is helping in fixing errors before I go too far into the development. I am learning data visualization and would not have easily come across all the tips without this book.
Learning IPython for Interactive Computing and Data Visualization
Unlock this book and the full library for FREE
Start free trial