Chapter 16: Packaging Python

Python is the most popular programming language for machine learning. Combine that with the proliferation of machine learning in our day-to-day lives and it is no surprise that the desire to run Python on edge devices is intensifying. Even in this era of transpilers and WebAssembly, packaging Python applications for deployment remains an unsolved problem. In this chapter, you will learn what choices are out there for bundling Python modules together and when to use one method over another.

We start with a look back at the origins of today's Python packaging solutions, from the built-in standard distutils to its successor, setuptools. Next, we examine the pip package manager, before moving on to venv for Python virtual environments, followed by conda, the reigning general-purpose cross-platform solution. Lastly, I will show you how to use Docker to bundle Python applications along with their user space environment for rapid deployment to the cloud...

Technical requirements

To follow along with the examples, make sure you have the following packages installed on your Linux-based host system:

• Python: Python 3 interpreter and standard library
• pip: Package installer for Python 3
• venv: Python module for creating and managing lightweight virtual environments
• Miniconda: Minimal installer for the conda package and virtual environment manager
• Docker: Tool for building, deploying, and running software inside containers

I recommend using Ubuntu 20.04 LTS or later for this chapter. Even though Ubuntu 20.04 LTS runs on the Raspberry Pi 4, I still prefer to develop on an x86-64 desktop PC or laptop. I choose Ubuntu for my development environment because the distribution maintainers keep Docker up to date. Ubuntu 20.04 LTS also comes with Python 3 and pip already installed since Python is used extensively throughout the system. Do not uninstall python3 or you will render Ubuntu unusable. To install venv on Ubuntu...

Retracing the origins of Python packaging

The Python packaging landscape is a vast graveyard of failed attempts and abandoned tools. Best practices around dependency management change often within the Python community and the recommended solution one year may be a broken nonstarter the next. As you research this topic, remember to look at when the information was published and do not trust any advice that may be out of date.

Most Python libraries are distributed using distutils or setuptools, including all the packages found on the Python Package Index (PyPI). Both distribution methods rely on a setup.py project specification file that the package installer for Python (pip) uses to install a package. pip can also generate or freeze a precise list of dependencies after
a project is installed. This optional requirements.txt file is used by pip in conjunction with setup.py to ensure that project installations are repeatable.

distutils

distutils is the original packaging system...

Installing Python packages with pip

You now know how to define your project's dependencies in a setup.py script. But how do you install those dependencies? How do you upgrade a dependency or replace it when you find a better one? How do you decide when it is safe to delete a dependency you no longer need? Managing project dependencies is a tricky business. Luckily, Python comes with a tool called pip that can help, especially in the early stages of your project.

The initial 1.0 release of pip arrived on April 4, 2011, around the same time that
Node.js and npm were taking off. Before it became pip, the tool was named pyinstall. pyinstall was created in 2008 as an alternative to easy_install, which came bundled with setuptools at the time. easy_install is now deprecated and setuptools recommends using pip instead.

Since pip is included with the Python installer and you can have multiple versions of Python installed on your system (for example, 2.7 and 3.8), it helps to know...

Managing Python virtual environments
with venv

A virtual environment is a self-contained directory tree containing a Python interpreter for a particular version of Python, a pip executable for managing project dependencies, and a local site-packages directory. Switching between virtual environments tricks the shell into thinking that the only Python and pip executables available are the ones present in the active virtual environment. Best practice dictates that you create a different virtual environment for each of your projects. This form of isolation solves the problem of two projects depending on different versions of the same package.

Virtual environments are not new to Python. The system-wide nature of Python installations necessitates them. Besides enabling you to install different versions of the same package, virtual environments also provide an easy way for you to run multiple versions of the Python interpreter. Several options exist for managing Python virtual environments...

Installing precompiled binaries with conda

conda is a package and virtual environment management system used by the Anaconda distribution of software for the PyData community. The Anaconda distribution includes Python as well as binaries for several hard-to-build open source projects such as PyTorch and TensorFlow. conda can be installed without the full Anaconda distribution, which is very large, or the minimal Miniconda distribution, which is still over 256 MB.

Even though it was created for Python shortly after pip, conda has evolved into
a general-purpose package manager like APT or Homebrew. Now, it can be used to package and distribute software for any language. Because conda downloads precompiled binaries, installing Python extension modules is a breeze. Another one of conda's
big selling points is that it is cross-platform, with full support for Linux, macOS,
and Windows.

Besides package management, conda is also a full-blown virtual environment manager. Conda virtual...

Deploying Python applications with Docker

Docker offers another way to bundle Python code with software written in other languages. The idea behind Docker is that instead of packaging and installing your application onto a preconfigured server environment, you build and ship a container image with your application and all its runtime dependencies. A container image is more like a virtual environment than a virtual machine. A virtual machine is a complete system image including a kernel and an operating system. A container image is a minimal user space environment that only comes with the binaries needed to run your application.

Virtual machines run on top of a hypervisor that emulates hardware. Containers run directly on top of the host operating system. Unlike virtual machines, containers are able to share the same operating system and kernel without the use of hardware emulation. Instead, they rely on two special features of the Linux kernel for isolation: namespaces and cgroups...

Summary

By now, you're probably asking yourself, what does any of this Python packaging stuff have to do with embedded Linux? The answer is not much, but bear in mind that the word programming also happens to be in the title of this book. And this chapter has everything to do with modern-day programming. To succeed as a developer in this day and age, you need to be able to deploy your code to production fast, frequently, and in a repeatable manner. That means managing your dependencies carefully and automating as much of the process as possible. You have now seen what tools are available for doing that with Python.

In the next chapter, we will look in detail at the Linux process model and describe what a process really is, how it relates to threads, how they cooperate, and how they are scheduled. Understanding these things is important if you want to create a robust and maintainable embedded system.

Further reading

The following resources have more information about the topics introduced in this chapter:

• Python Packaging User Guide, PyPA: https://packaging.python.org
• setup.py vs requirements.txt, by Donald Stufft: https://caremad.io/posts/2013/07/setup-vs-requirement
• pip User Guide, PyPA: https://pip.pypa.io/en/latest/user_guide/
• Poetry Documentation, Poetry: https://python-poetry.org/docs
• conda user guide, Continuum Analytics: https://docs.conda.io/projects/conda/en/latest/user-guide
• docker docs, Docker Inc.: https://docs.docker.com/engine/reference/commandline/docker