Chapter 3. Extending Applications

As mentioned in previous chapters, I want to show you how to interact or extend existing code using Cython. So, let's get right to doing that. Cython was originally designed to make raw Python computation faster. So, the initial proof of concept for Cython was to enable programmers to take existing Python code and use Cython's cdef keyword to require native typing to bypass the Python runtime for heavy computation. The culmination of this is increased performance in the time it takes to perform calculations and lower memory usage. It's even possible to write type-safe wrappers to existing Python libraries for fully typed Python code.

In this chapter, we will first see an example of typing Python code. Next, I will demonstrate the Cython cdef class, which allow us to wrap native C/C++ types into garbage collected Python classes. We will also see how to extend the native application Tmux with Python code by creating a pure Python command object, which is directly...

Cython pure Python code

Let's view a mathematical application that is actually taken from the Cython documentation. I wrote this equivalent in pure Python so that we can compare the speed. If you open the primes example for this chapter, you will see two programs—the Cython primes.pyx example, and my pure Python port. They both look almost the same:

def primes(kmax):
    n = 0
    k = 0
    i = 0
    if kmax > 1000:
        kmax = 1000
    p = [0] * kmax
    result = []
    k = 0
    n = 2
    while k < kmax:
        i = 0
        while i < k and n % p[i] != 0:
            i = i + 1
        if i == k:
            p[k] = n
            k = k + 1
            result.append(n)
        n = n + 1
    return result
primes (10000)

This really is a direct Python port of that Cython code. Both call primes (10000), but the evaluation time is very different between them in terms of performance:

$ make
cython --embed primes.pyx
gcc -g -O2 -c primes.c -o primes.o `python-config --includes`
gcc ...

Compiling pure Python code

Another use for Cython is to compile Python code. For example, if we go back to the primes example, we can do the following:

$ cython pyprimes.py –embed
$ gcc -g -O2 pyprimes.c -o pyprimes `python-config --includes –libs`

Then, we can compare the three different versions of the same program: the Cython version using cdef for native types, the pure Python version running as a Python script, and finally, the Cython-compiled pure Python version, which results in an executable binary of Python code:

• First, the Cython version using native types:

  $ time ./primes
  real    0m0.050s
  user    0m0.035s
  sys     0m0.013s
  

• Next, the executable pure Python version:

  $ time ./pyprimes
  real    0m0.139s
  user    0m0.122s
  sys     0m0.013s
  

• And finally, the Python script version:

  philips-macbook:primes redbrain$ time python pyprimes.py
  real    0m0.184s
  user    0m0.165s
  sys     0m0.016s
  

The pure Python version runs the slowest, the compiled Python version runs a little bit faster, and finally...

Python garbage collector

When wrapping up native structs, for example, it can be very tempting to follow standard C/C++ idioms and require the Python programmer to call, allocate, and release manually on different objects. This is very tedious and not very Pythonic. Cython allows us to create cdef classes, which have extra hooks for initialization and deallocation that we can use to control all memory management of structs. These hooks are triggered automatically by the Python garbage collector, making everything nice and simple. Consider the following simple struct:

typedef struct data {
  int value;
} data_t;

We can then write the Cython declaration of the C struct into PyData.pxd as follows:

cdef extern from "Data.h":
    struct data:
        int value
    ctypedef data data_t

Now that we have defined the struct, we can wrap up the struct into a class:

cimport PyData

cdef class Data(object):
    cdef PyData.data_t * _nativeData
    …

Wrapping up data into a class like this will require us...

Extending Tmux

Tmux is a terminal multiplexer inspired by GNU Screen (http://tmux.github.io/), but it supports much simpler and better configuration. More importantly, the implementation is much cleaner and easier to maintain, and it also uses libevent and very well-written C code.

I want to show you how you can extend Tmux with new built-in commands by writing Python code instead of C. Overall, there are several parts to this project, as follows:

• Hack the autotool's build system to compile in Cython

• Create PXD declarations to the relevant declarations such as struct cmd_entry

• Embed Python into Tmux

• Add the Python command to the global Tmux cmd_table

Let's take a quick look at the Tmux source, and in particular any of the cmd-*.c files that contain command declarations and implementations. Consider, for example, that cmd-kill-window.c is the command entry. This tells Tmux the name of the command, its alias, and how it may or may not accept arguments; finally, it accepts a function pointer to the...

Embedding Python

Now that we have files being compiled, we need to initialize Python. Our module. Tmux is a forked server that clients connect to, so try not to think of it as a single-threaded system. It's a client and a server, so all commands are executed on the server. Now, let's find where the event loop is started in the server, and initialize and finalize the server here so that it's done correctly. Looking at int server_start(int lockfd, char *lockfile), we can add the following:

#ifdef HAVE_PYTHON
  Py_InitializeEx (0);
#endif
  server_loop();
#ifdef HAVE_PYTHON
  Py_Finalize ();
#endif

Python is now embedded into the Tmux server. Notice that instead of using simply Py_Initialize, I used Py_InitializeEx (0). This replicates the same behavior, but doesn't start up normal Python signal handlers. Tmux has its own signal handlers, so I don't want to override them. It's probably a good idea when extending established applications such as this to use Py_InitializeEx (0), since they generally...

Cythonizing struct cmd_entry

Next, let's consider creating a cythonfile.pxd file for the necessary cdef declarations of Tmux that we need to be aware of. We need to look at the struct cmd_entry declaration, and work backwards from this:

struct cmd_entry {
  const char  *name;
  const char  *alias;

  const char  *args_template;
  int     args_lower;
  int     args_upper;

  const char  *usage;
  int     flags;

  void     (*key_binding)(struct cmd *, int);
  int     (*check)(struct args *);
  enum cmd_retval   (*execc)(struct cmd *, struct cmd_q *);
};

As you can see, cmd_entry depends on several other types, so we need to work backwards a little bit. If you're going to be lazy and live dangerously, you can get away with it sometimes if you don't care about accessing the data correctly by casting any pointers such as void *. But if you're a seasoned C programmer, you know this is fairly dangerous and should be avoided. You can see this type depends on struct cmd *, struct cmd_q *, and struct...

Implementing a Tmux command

One caveat with Cython is that we cannot statically initialize structs like we can in C, so we need to make a hook so that we can initialize cmd_entry on Python startup:

cimport cmdpython

cdef public cmd_entry cmd_entry_python

With this, we now have a public declaration of cmd_entry_python, which we will initialize in a startup hook as follows:

cdef public void tmux_init_cython () with gil:
    cmd_entry_python.name = "python"
    cmd_entry_python.alias = "py"
    cmd_entry_python.args_template = ""
    cmd_entry_python.args_lower = 0
    cmd_entry_python.args_upper = 0
    cmd_entry_python.usage = "python usage..."
    cmd_entry_python.flags = 0
    #cmd_entry_python.key_binding = NULL
    #cmd_entry_python.check = NULL
    cmd_entry_python.execc = python_exec

Remember that because we declared this in the top level, we know it's on the heap and don't need to declare any memory to the structure, which is very handy for us. You've seen struct access before; the function...

Hooking everything together

We now have to fiddle with Tmux just a tiny bit more, but it's fairly painless, and once we are done we are free to be creative. Fundamentally, we should call the cmd_entry initialization hook in server.c just before we forget about it:

#ifdef HAVE_PYTHON
  Py_InitializeEx (0);
  tmux_init_cython ();
#endif

  server_loop();

#ifdef HAVE_PYTHON
  Py_Finalize ();
#endif

Now that this is done, we need to make sure we add the cmd_entry_python extern declaration to tmux.h:

extern const struct cmd_entry cmd_wait_for_entry;
#ifdef HAVE_PYTHON
# include "cmdpython.h"
#endif

Finally, add this to cmd_table:

const struct cmd_entry *cmd_table[] = {
  &cmd_attach_session_entry,
  &cmd_bind_key_entry,
  &cmd_break_pane_entry,
…
  &cmd_wait_for_entry,
  &cmd_entry_python,
  NULL
};

Now that this is done, I think we're good to go—let's test out this baby. Compile Tmux with the following:

$ ./configure –enable-python
$ make
$ ./tmux -vvv
$ tmux: C-b :python
$ tmux...

Summary

There are many different techniques and ideas demonstrated in this chapter, but it should serve as a strong reference on common techniques. We saw the speedups in using native types to bypass the runtime, and compiled Python code into its own binary. The pyximport statement shows us we can bypass compilation and simply import .pyx files as if it was normal Python. Finally, I ended the chapter with a step-by-step demonstration of my process in embedding Python into Tmux. In the next chapter, we will see debugging in action using gdb, and some caveats in using Cython.