Chapter 1. Winding Along with IPython

In this chapter, we will cover the following topics:

Installing IPython
Using IPython as a shell
Reading manual pages
Installing Matplotlib
Running a web notebook
Exporting a web notebook
Importing a web notebook
Configuring a notebook server
Exploring the SymPy profile

Introduction

IPython, which is available at http://ipython.org/, is a free, open source project available for Linux, Unix, Mac OS X, and Windows. The IPython authors only request that you cite IPython in any scientific work where IPython was used. It provides the following components, among others:

Interactive Python shells (terminal-based and Qt application)
A web notebook (available in IPython 0.12 and later) with support for rich media and plotting
IPython is compatible with Python versions 2.5, 2.6, 2.7, 3.1, and 3.2

You can try IPython in cloud without installing it on your system, by going to the following URL: http://www.pythonanywhere.com/try-ipython/. There is a slight delay compared to locally installed software; so this is not as good as the real thing. However, most of the features available in the IPython interactive shell seem to be available. They also have a Vi (m) editor, which if you like vi, is of course great. You can save and edit files from your IPython sessions. The author of this book doesn't care much about other editors, such as the one that starts with E and ends with macs. This should, however, not be a problem.

Installing IPython

IPython can be installed in various ways depending on your operating system. For the terminal-based shell, there is a dependency on readline. The web notebook requires tornado and zmq.

In addition to installing IPython, we will install setuptools, which gives you the easy_install command. The easy_install command is the default, standard package manager for Python. pip can be installed once you have easy_install available. The pip command is similar to easy_install, and adds options such as uninstalling.

How to do it...

This section describes how IPython can be installed on Windows, Mac OS X, and Linux. It also describes how to install IPython and its dependencies with easy_install and pip, or from source.

Installing IPython and setup tools on Windows: A binary Windows installer for Python 2 or Python 3 is available on the IPython website. Also see http://ipython.org/ipython-doc/stable/install/install.html#windows.
Install setuptools with an installer from http://pypi.python.org/pypi/setuptools#files. Then install pip; for instance:
```
cd C:\Python27\scripts
python .\easy_install-27-script.py pip
```
Installing IPython On Mac OS X: Install the Apple Developer Tools (Xcode) if necessary. Xcode can be found on the OSX DVDs that came with your Mac or App Store. Follow the easy_install/pip instructions, or the installing from source instructions provided later in this section.
Installing IPython On Linux: Because there are so many Linux distributions, this section will not be exhaustive.
- On Debian, type the following command:
```
su – aptitude install ipython python-setuptools
```
- On Fedora, the magic command is as follows:
```
su – yum install ipython python-setuptools-devel
```
- The following command will install IPython on Gentoo:
```
su – emerge ipython
```
- For Ubuntu, the install command is as follows:
```
sudo apt-get install ipython python-setuptools
```
Installing IPython with easy_install or pip: Install IPython and all the dependencies required for the recipes in this chapter with easy_install, using the following command:
```
easy_install ipython pyzmq tornado readline
```
Alternatively, you can first install pip with easy_install, by typing the following command in your terminal:
```
easy_install pip
```
After that, install IPython using pip, with the following command:
```
sudo pip install ipython pyzmq tornado readline
```
Installing from source: If you want to use the bleeding edge development version, then installing from source is for you.
1. Download the latest tarball from https://github.com/ipython/ipython/downloads.
2. Unpack the source code from the archive:
```
tar xzf ipython-<version>.tar.gz
```
3. If you have Git installed, you can clone the Git repository instead:
```
$ git clone https://github.com/ipython/ipython.git
```
4. Go to the ipython directory:
```
cd ipython
```
5. Run the setup script. This may require you to run the command with sudo, as follows:
```
sudo setup.py install
```

How it works...

We installed IPython using several methods. Most of these methods install the latest stable release, except when you install from source, which will install the development version.

Using IPython as a shell

Scientists and engineers are used to experimenting. IPython was created by scientists with experimentation in mind. The interactive environment that IPython provides is viewed by many as a direct answer to Matlab, Mathematica, and Maple and R.

Following is a list of features of the IPython shell:

Tab completion
History mechanism
Inline editing
Ability to call external Python scripts with %run
Access to system commands
The pylab switch
Access to Python debugger and profiler

How to do it...

This section describes how to use the IPython shell.

The pylab switch: The pylab switch automatically imports all the Scipy, NumPy, and Matplotlib packages. Without this switch, we would have to import these packages ourselves.

All we need to do is enter the following instruction on the command line:

$ ipython -pylab
Type "copyright", "credits" or "license" for more information.

IPython 0.12 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

Welcome to pylab, a matplotlib-based Python environment [backend: MacOSX].
For more information, type 'help(pylab)'.
In [1]: quit()
quit() or Ctrl 
+ D quits the IPython shell.

Saving a session: We might want to be able to go back to our experiments. In IPython, it is easy to save a session for later use, with the following command:

In [1]: %logstart
Activating auto-logging. Current session state plus future input saved.
Filename       : ipython_log.py
Mode           : rotate
Output logging : False
Raw input log  : False
Timestamping   : False
State          : active

Logging can be switched off as follows:

In [9]: %logoff
Switching logging OFF

Executing system shell commands: Execute system shell commands in the default IPython profile by prefixing the command with the ! symbol. For instance, the following input will get the current date:
```
In [1]: !date
```
In fact, any line prefixed with ! is sent to the system shell. Also, we can store the command output, as shown here:
```
In [2]: thedate = !date
In [3]: thedate
```
Displaying history: We can show the history of commands with the %hist command () for example:
```
In [1]: a = 2 + 2

In [2]: a
Out[2]: 4

In [3]: %hist
a = 2 + 2
a
%hist
```
This is a common feature in Command Line Interface (CLI) environments. We can also search through the history with the -g switch
```
In [5]: %hist -g a = 2
    1: a = 2 + 2
```

Note

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

How it works...

We saw a number of so called "magic functions" in action. These functions start with the % character. If the magic function is used on a line by itself, the % prefix is optional.

Reading manual pages

When we are in IPython's pylab mode, we can open manual pages for NumPy functions with the help command. It is not necessary to know the name of a function. We can type a few characters and then let tab completion do its work. Let's, for instance, browse the available information for the arange function.

How to do it...

We can browse the available information, in either of the following two ways:

Calling the help function: Call the help command. Type a few characters of the function and press the Tab key:
Querying with a question mark: Another option is to put a question mark behind the function name. You will then, of course, need to know the function name, but you don't have to type help:
```
In [3]: arange?
```

How it works...

Tab completion is dependent on readline, so you need to make sure it is installed. The question mark gives you information from docstrings.

Installing Matplotlib

Matplotlib is a very useful plotting library, which we will need for the next recipe. It depends on NumPy, but in all likelihood you already have NumPy installed.

How to do it...

We will see how Matplotlib can be installed in Windows, Linux, and Mac, and also how to install it from source.

Installing Matplotlib on Windows: Install with the Enthought distribution (http://www.enthought.com/products/epd.php).
It might be necessary to put the msvcp71.dll file in your C:\Windows\system32 directory. You can get it from http://www.dll-files.com/dllindex/dll-files.shtml?msvcp71.
Installing Matplotlib on Linux: Let's see how Matplotlib can be installed in the various distributions of Linux:
- The install command on Debian and Ubuntu is as follows:
```
sudo apt-get install python-matplotlib
```
- The install command on Fedora/Redhat is as follows:
```
su - yum install python-matplotlib
```
Installing from source: Download the latest source from the tar.gz release at Sourceforge (http://sourceforge.net/projects/matplotlib/files/) or from the Git repository using the following command:
```
git clone git://github.com/matplotlib/matplotlib.git
```
Once it has been downloaded, build and install as usual with the following command:
```
cd matplotlib
python setup.py install
```
Installing Matplotlib on Mac: Get the latest DMG file from http://sourceforge.net/projects/matplotlib/files/matplotlib/, and install it.

Running a web notebook

The newest release of IPython introduced a new exciting feature – the web notebook. A so called "notebook server" can serve notebooks over the web. We can now start a notebook server and have a web-based IPython environment. This environment has most of the features in the regular IPython environment. The new features include the following:

Displaying images and inline plots
Using HTML and Markdown in text cells
Importing and exporting of notebooks

Getting ready

Before we start, we should make sure that all the required software is installed. There is a dependency on tornado and zmq. See the Installing IPython recipe in this chapter for more information.

How to do it...

Running a notebook: We can start a notebook with the following code:
```
$ ipython notebook

[NotebookApp] Using existing profile dir: u'/Users/ivanidris/.ipython/profile_default'
[NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888
[NotebookApp] Use Control-C to stop this server and shut down all kernels.
```
As you can see, we are using the default profile. A server started on the local machine at port 8888. We will learn how to configure these settings later on in this chapter. The notebook is opened in your default browser; this is configurable as well:
IPython lists all the notebooks in the directory where you started the notebook. In this example no notebooks were found. The server can be stopped with Ctrl + C.
Running a notebook in the pylab mode: Run a web notebook in the pylab mode with the following command:
```
$ ipython notebook --pylab
```
This loads the Scipy, NumPy, and Matplotlib modules.
Running notebook with inline figures: We can display inline Matplotlib plots with the inline directive, using the following command:
```
$ ipython notebook --pylab inline
```

Create a notebook: Click on the New Notebook button to create a new notebook:
Create an array: Create an array with the arange function. Type the command in the following screenshot, and press Enter:
Next, enter the following command and press Enter. You will see the output as shown in Out [2] in the following screenshot:
Plot the sinc function: Apply the sinc function to the array and plot the result, as shown in the following screenshot:

How it works...

The inline option lets you display inline Matplotlib plots. When combined with the pylab mode, you don't need to import the NumPy, SciPy, and Matplotlib packages.

Exporting a web notebook

Sometimes you will want to exchange notebooks with friends or colleagues. The web notebook provides several methods to export your data.

How to do it...

A web notebook can be exported using the following options:

The Print option: The Print button doesn't actually print the notebook, but allows you to export the notebook as PDF or HTML document.

Downloading the notebook: Download your notebook to a location chosen by you, using the Download button. We can specify whether we want to download the notebook as .py file, which is just a normal Python program, or in the JSON format as a .ipynb file. The notebook we created in the previous recipe looks like the following, after exporting:

{
 "metadata": {
  "name": "Untitled1"
 }, 
 "nbformat": 2, 
 "worksheets": [
  {
    "cells": [
    {
      "cell_type": "code", 
      "collapsed": false, 
      "input": [
        "plot(sinc(a))"
      ], 
      "language": "python", 
      "outputs": [
      {
        "output_type": "pyout", 
        "prompt_number": 3, 
        "text": [
          "[&lt;matplotlib.lines.Line2D at 0x103d9c690&gt;]"
        ]
      }, 
      {
        "output_type": "display_data", 
        "png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD9CAYAAABZVQdHAAAABHNCSVQICAgIf...
          mgkAAAAASUVORK5CYII=\n"
      }
      ], 
      "prompt_number": 3
    }
    ]
  }
  ]
}

Note

Some of the text has been omitted for brevity. This file is not intended for editing or reading even, but it is pretty readable if you ignore the image representation part. For more information about JSON please see https://en.wikipedia.org/wiki/JSON.

Saving the notebook: Save the notebook using the Save button. This will automatically export a notebook in the native JSON .ipynb format. The file will be stored in the directory where you started IPython initially.

Importing a web notebook

Python scripts can be imported as a web notebook. Obviously, we can also import previously exported notebooks.

How to do it...

The following steps show how a python script can be imported as a web notebook:

Import a python script by dragging it from Explorer or Finder into the notebook page. The following screenshot is an example of what we see after dragging the vectorsum.py from NumPy Beginner's Guide into the notebook page:
Click the Upload button to import the program. IPython does a decent job of importing the code. Unfortunately, as shown in the following screenshot, the code is all placed in one cell. At least that is how it worked at the time of writing:

Tag the script for multiple cells.

In order to split the code into multiple cells we need to use special tags. These tags are in fact Python comments, but they look a bit like XML tags. The code has to start with the following tag:

# <nbformat>2</nbformat>

This indicates the format of the notebook. Each new code cell is indicated with the following tag:

# <codecell>

The following is the tagged code:

# <nbformat>2</nbformat>
#!/usr/bin/env/python

from datetime import datetime
import numpy
"""
 Chapter 1 of NumPy Beginners Guide.
 This program demonstrates vector addition the Python way.
 Run from the command line as follows
     
  python vectorsum.py n
 
 where n is an integer that specifies the size of the vectors.

 The first vector to be added contains the squares of 0 up to n. 
 The second vector contains the cubes of 0 up to n.
 The program prints the last 2 elements of the sum and the elapsed time.
"""

def numpysum(n):
   a = numpy.arange(n) ** 2
   b = numpy.arange(n) ** 3
   c = a + b

   return c

def pythonsum(n):
   a = range(n)
   b = range(n)
   c = []

   for i in range(len(a)):
       a[i] = i ** 2
       b[i] = i ** 3
       c.append(a[i] + b[i])

   return c
   
# <codecell>
size = int(50)

# <codecell>
start = datetime.now()
c = pythonsum(size)
delta = datetime.now() - start
print "The last 2 elements of the sum", c[-2:]
print "PythonSum elapsed time in microseconds", delta.microseconds

# <codecell>
start = datetime.now()
c = numpysum(size)
delta = datetime.now() - start
print "The last 2 elements of the sum", c[-2:]
print "NumPySum elapsed time in microseconds", delta.microseconds

The code is split into several cells according to the tags, as shown in the following screenshot:

Configuring a notebook server

A public notebook server needs to be secure. You should set a password and use a SSL certificate to connect to it. We need the certificate to provide secure communication over https (for more information see https://en.wikipedia.org/wiki/Transport_Layer_Security).

How to do it...

The following steps describe how to configure a secure notebook server:

Generate a password: We can generate a password from IPython. Start a new IPython session, and type in the following commands:
```
In [1]: from IPython.lib import passwd

In [2]: passwd()
Enter password: 
Verify password: 
Out[2]: 'sha1:0e422dfccef2:84cfbcbb3ef95872fb8e23be3999c123f862d856'
```
At the second input line you will be prompted for a password. You need to remember this password. A long string is generated. Copy this string because we will need it later on.

Create a SSL certificate: To create a SSL certificate, you will need to have the openssl command in your path.

Setting up the openssl command is not rocket science, but can be tricky. Unfortunately, it is outside the scope of this book. On the bright side there are plenty of tutorials available online to help you further.

Execute the following command to create a certificate with the name mycert.pem:

$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
Generating a 1024 bit RSA private key
......++++++
........................++++++
writing new private key to 'mycert.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:
Email Address []:

The openssl utility prompts you to fill in some fields. For more information, check the relevant man page (short for manual page).

Create a server profile: Create a special profile for the server using the following command:
```
ipython profile create nbserver
```
Edit the profile configuration file: Edit the configuration file. In this example, it can be found in Edit in ~/.ipython/profile_nbserver/ipython_notebook_config.py.
The configuration file is pretty large, so we will omit many of the lines in it. The lines that we need to change at minimum are:
```
c.NotebookApp.certfile = u'/absolute/path/to/your/certificate'
c.NotebookApp.password = u'sha1:b...your password'
c.NotebookApp.port = 9999
```
Notice that we are pointing to the SSL certificate we created. We set a password and changed the port to 9999.
Start the server: Using the following command, start the server to check whether the changes worked.
```
ipython notebook --profile=nbserver
[NotebookApp] Using existing profile dir: u'/Users/ivanidris/.ipython/profile_nbserver'
[NotebookApp] The IPython Notebook is running at: https://127.0.0.1:9999
[NotebookApp] Use Control-C to stop this server and shut down all kernels.
```
The server is running on port 9999, and you need to connect to it via https. If everything goes well, we should see a login page. Also, you would probably need to accept a security exception in your browser.

How it works...

We created a special profile for our public server. There are some sample profiles that are already present, such as the default profile. Creating a profile adds a profile_<profilename> folder to the .ipython directory with, among others, a configuration file. The profile can then be loaded with the --profile=<profile_name> command-line option. We can list the profiles with the following command:

ipython profile list

Available profiles in IPython:
    cluster
    math
    pysh
    python3

    The first request for a bundled profile will copy it
    into your IPython directory (/Users/ivanidris/.ipython),
    where you can customize it.

Available profiles in /Users/ivanidris/.ipython:
    default
    nbserver
    sh

Exploring the SymPy profile

IPython has a sample SymPy profile. SymPy is a Python symbolic, mathematics library. For instance, we can simplify algebraic expressions or differentiate, similar to Mathematica and Maple. SymPy is obviously a fun piece of software, but is not directly necessary for our journey through the NumPy landscape. Consider this as an optional bonus recipe. Like dessert, feel free to skip, although you might miss out on the sweetest piece of this chapter.

Getting ready

Install SymPy using either easy_install, or pip:

easy_install sympy
sudo pip install sympy

How to do it...

Look at the configuration file, which can be found at ~/.ipython/profile_sympy/ipython_config.py. The contents are as follows:

c = get_config()
app = c.InteractiveShellApp

# This can be used at any point in a config file to load a sub config
# and merge it into the current one.
load_subconfig('ipython_config.py', profile='default')

lines = """
from __future__ import division
from sympy import *
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)
"""

# You have to make sure that attributes that are containers already
# exist before using them.  Simple assigning a new list will override
# all previous values.
if hasattr(app, 'exec_lines'):
    app.exec_lines.append(lines)
else:
    app.exec_lines = [lines]

# Load the sympy_printing extension to enable nice printing of sympy expr's.
if hasattr(app, 'extensions'):
    app.extensions.append('sympyprinting')
else:
    app.extensions = ['sympyprinting']

This code accomplishes the following:

Loading the default profile
Importing the SymPy packages
Defining symbols

Start IPython with the SymPy profile using the following command:
```
ipython --profile=sympy
```
Expand an algebraic expression using the command shown in the following screenshot: