Chapter 2. Understanding Cython

As I mentioned previously, there are a number of methods of using Cython. As the basics are very familiar to any Python programmer, it's important to review the linking models before getting into the programming language. This is what drives the design of applications when using Cython.

Next, we will get more familiar with the Cython programming language constructs, namely, the distinction between cdef and cpdef. Then, we will look at getting the most out of Cython by interfacing directly with native C types. Later in this book, we will see that it's possible to use native C++ STL container types. This is where you will gain the optimizations in execution, as no Python runtime is required to work with native types.

Finally, we will see how easy it is to work with callbacks to and from C and Python code. This is an interesting technique whereby you can offload logic from C code to Python.

Therefore, in this chapter, we will be diving into the following topics...

Linking models

Linking models are extremely important when considering how we can extend or embed things in native applications. There are two main linking models for Cython:

Fully embedded Python within C/C++ code, which looks like the following screenshot:

Using this method of embedding the Python runtime into a native application means you initiate execution of code directly from any point in your C/C++ code, as opposed to the Chapter 1, Cython Won't Bite where we had to run the Python interpreter and call an import to execute native code.

For the sake of completeness, here is the import model of using Cython:

This would be a more Pythonic approach to Cython, and will be helpful if your code base is mostly Python. We will review an example of the Python lxml module, which provides a Cython backend, later in this book, and we can compare it to the native Python backend to review the speed and execution of both code bases to perform the same task.

Cython keyword – cdef

The cdef keyword tells the compiler that this statement is a native C type or native function. Remember from Chapter 1, Cython Won't Bite that we used this line to declare the C prototype function:

cdef int AddFunction(int, int)

This is the line that let us wrap the native C function into a Python callable using the Python def keyword. We can use this in many contexts, for example, we can declare normal variables for use within a function to speed up execution:

def square(int x):
    return x ** 2

This is a trivial example, but it will tell the compiler that we will always be squaring an integer. However, for normal Python code, it's a little more complicated as Python has to worry a lot more about losing precision when it comes to handling many different types. But in this case, we know exactly what the type is and how it can be handled.

You might also have noticed that this is a simple def function, but because it will be fed to the Cython compiler, this will work just...

Typedef and function pointers

The typedef in C/C++ code allows the programmer to give a new name or alias to any type. For example, one could typedef an int to myint. Or you can just simply typedef a struct so that you don't have to refer to the struct with the keyword struct every time. For example, consider this C struct and typedef:

struct foobar {
  int x;
  char * y;
};
typedef struct foobar foobar_t;

In Cython, this can be described by the following:

cdef struct foobar:
    int x
    char * y
ctypedef foobar foobar_t

Note we can also typedef pointer types as below:

ctypedef int * int_ptr

We can also typedef function C/C++ pointers, as follows:

typedef void (*cfptr) (int)

In Cython, this will be as follows:

ctypedef void (*cfptr)(int)

Using the function pointer is just as you would expect:

cdef cfptr myfunctionptr = &myfunc

There is some magic going on here with function pointers as it's simply not safe for raw Python code to directly call a Python function or vice versa. Cython understands...

The public keyword

This is a very powerful keyword in Cython. It allows any cdef declaration with the public modifier to output a respective C/C++ header with the relative declaration accessible from C/C++. For example, we can declare:

cdef public struct CythonStruct:
    size_t number_of_elements;
    char ** elements;

Once the compiler handles this, you will have an output of cython_input.h:

 struct CythonStruct {
    size_t number_of_elements;
    char ** elements;
};

The main caveat, if you're going to call the Python public declarations directly from C, is that, if your link model is fully embedded and linked against libpython.so, you need to use some boilerplate code to initialize Python correctly:

#include <Python.h>

int main(int argc, char **argv) {
    Py_Initialize ();
    // code in here
    Py_Finalize ();
    return 0;
}

And before calling anything with the function, you need to initialize the Python module example if you have a cythonfile.pyx file, and compile it with the...

Keyword cpdef

So far, we have seen two different function declarations in Cython, def and cdef, to define functions. There is one more declaration—cpdef. The def is a Python-only function, so it is only callable from Python or Cython code blocks; calling from C does not work. The cdef is the opposite; this means that it's callable from C and not from Python. For example, if we create a function such as:

cpdef public test (int x):
   …
  return 1

It will generate the following function prototype:

__PYX_EXTERN_C DL_IMPORT(PyObject) *test(int, int __pyx_skip_dispatch);

The public keyword will make sure we generate the needed header so that we can call it from C. Calling from pure Python, we can work with this as if it was just any other Python function. The drawback of using cpdef is that the native return type is PyObject *, which requires you to know exactly what the return type is and consult the Python API documentation to access the data. I prefer keeping bindings between the languages simpler...

Logging from C/C++ into Python

An example of everything brought together is reusing the Python logging module directly from C. We want a few macros, such as info, error, and debug that can all handle a variable number of arguments and works as if we are calling a simple printf method.

To achieve this, we must make a Python logging backend for our C/C++ code. We need an initialization function to tell Python about our output logfile, and some wrappers for each info, error, and debug. We can simply write the public cdef wrappers as:

import logging

cdef public void initLoggingWithLogFile(const char * logfile):
    logging.basicConfig(filename = logfile,
                        level = logging.DEBUG,
                        format = '%(levelname)s %(asctime)s: %(message)s',
                        datefmt = '%m/%d/%Y %I:%M:%S')

cdef public void python_info(char * message):
    logging.info(message)

cdef public void python_debug(char * message):
    logging.debug(message)

cdef public void...

Using Python ConfigParser from C/C++

I really like Python's ConfigParser API. I find using an INI style config file to be very readable and nice to work with as opposed to using XML or JSON. There are very few cross-platform libraries available to do this. However, when you have Cython, all you need is Python.

For this example, we will create a sample INI configuration file and write a simple API to access a list of sections, list of keys available in a section, and a way to get the value from a specified key within a section. These three functions will allow a programmer to access any INI file.

A sample INI file could be:

[example]
number = 15
path = some/path/to/something

[another_section]
test = something

An INI file is comprised of sections within the square brackets, followed by keys and values. It's a very simple way of doing configuration. Python's API allows for variables and substitution depending on the flavor of the ConfigParser. Firstly, we need a way to query the list of sections...

Callbacks from Python to C/C++

Callbacks are used extensively in asynchronous systems. Libraries such as libevent provide a powerful asynchronous core to process events. Let's build an example to set a C function as a callback into a Python backend, which will notify back again into the C code. Firstly, we will declare a public callback function typedef:

cdef public:
    ctypedef void (*callback)(int)

This will output a callback typedef. Next, we can declare a global callback on the stack:

cdef callback GlobalCallback

Once this is set, we can then notify the callback easily. Next, we need a way to set the callback and another to call the callback:

cdef public void SetCallback(callback cb):
    global GlobalCallback
    GlobalCallback = cb

Notice the global keyword from Python through which the compiler knows to use the global keyword and not create a temporary instance from within that suite:

cdef public void Notify(int value):
    global GlobalCallback
    if GlobalCallback != <callback>...

Cython PXD

The use of PXD files is very similar to that of header files in C/C++. When writing bindings to any C/C++ code, it is a good practice to declare all C/C++ interfaces within a .pxd file. This stands for Python External Declarations, at least it does in my mind. So, when we add blocks such as this:

cdef extern from "AddFunction.h":
    cdef int AddFunction(int, int)

We can instead put this directly into a bindings.pxd file and import this at any time inside any .pyx file:

cimport bindings

Notice the distinction between cimport for the .pxd files and a simple import for all normal Python imports.

Tip

Cython's input filenames cannot handle dashes (-) in their filenames. It's best to try and use CamelCase, since you can't use cimport my-import in Python.

Integration with build systems

This topic is basically dependent on the linking model that you choose if you are to choose the shared-library approach. I would recommend using Python distutils and if you are going for embedded Python, and if you like GNU or autotools, this section gives an example you can use.

Python Distutils

When compiling a native Python module, we can use distutils and cythonize inside our Setup.py build. It's the preferred way in Python to use Cython as part of the build:

from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("sourcecode.pyx")
)

This build file will support whichever version of Python you invoke the script with. When you run the build, your output will be of the same name of the input source code as a shared module in this case sourcecode.so.

GNU/Autotools

To embed Python code within C/C++ applications using the autotools build system the following snippet will help you. It will use python-config to get the...

Summary

There are a lot of fundamentals of using Cython in this chapter. It's important to review what you want to achieve when using Cython, since the different ways in which it can be used affects how you design a solution. We investigated the differences between def, cdef, and cpdef. We created public C/C++ declarations of types and callable functions. Using these public declarations, we showed how Python can callback into C code. For me, reusing any Python module within native code is very useful and interesting. I demonstrated how I use the Python logging and ConfigParser modules from C code. Appreciating these simple examples, we will see how we can extend C/C++ projects with Python code in the next chapter.