An important yet often overlooked aspect of Python programming is how to correctly set up and maintain a Python project and its runtime environment. It is often overlooked because it presents as an optional step for the Python ecosystem. And while this might be fine for learning Python language fundamentals, it can quickly become a problem for more complex projects where we need to maintain separate code bases and dependencies to ensure our projects do not interfere with one another, or worse as we will discuss, break operating system tools and utilities.

So, before we jump into IoT code and examples in later chapters, it is so very important for us to cover the steps required to set up a Python project and its run time environment.

In this chapter, we will cover the following topics:

• Understanding your Python installation
• Setting up a Python virtual environment
• Installing Python GPIO packages with pip
• Alternative methods of executing a Python script
• Raspberry Pi GPIO interface configuration

To perform the hands-0n exercises in this chapter, you will need the following:

• Raspberry Pi 4 Model B
• Raspbian OS Buster (with desktop and recommended software)
• Minimum Python version 3.5

These requirements are what the code examples in this book are based on. It's reasonable to expect that the code examples should work without modification on a Raspberry Pi 3 Model B or a different version of Raspbian OS as long as your Python version is 3.5 or higher.

The full source code for this book can be found on GitHub at the following URL: https://github.com/PacktPublishing/Practical-Python-Programming-for-IoT. We will clone this repository shortly when we come to the Setting up a Python virtual environment section.

In this section, we will find out which versions of Python you have installed on your Raspberry Pi. As we will discover, there are two versions of Python that come pre-installed on Raspbian OS. Unix-based operating systems (such as Raspbian OS) typically have Python version 2 and 3 pre-installed because there are operating-system-level utilities built with Python.

To find out which versions of Python you have on your Raspberry Pi, follow these steps:

1. Open a new Terminal and execute the python --version command:

$ python --version
Python 2.7.16

In my example, we see that Python version 2.7.16 has been installed.

2. Next, run the python3 --version command:

$ python3 --version
Python 3.7.3

In my example, we see that the second version of Python (that is, python3, with the 3) that is installed is version 3.7.3.

Don't worry if the minor versions (the numbers .7.16 after the 2 and .7.3 after 3) are not the same; it is the major versions 2 and 3 that are of interest. Python 2 is a legacy version of Python, while Python 3 is the current and supported version of Python at the time of writing. When we are starting a new Python development, we will practically always use Python 3 unless there are legacy issues we need to contend with.

If you are an experienced Python programmer, you may be able to discern whether a script is written for Python 2 or 3, but it's not always obvious by simply looking at a piece of code. Many new-to-Python developers experience frustrations by mixing up Python programs and code fragments that are meant for different Python versions. Always remember that code written for Python 2 is not guaranteed to be upward-comparable with Python 3 without modification.

A quick tip I can share to visually help to determine which Python version a code fragment is written for (if the programmer has not made it clear in the code comments) is to look for a print statement.

If you look at the following example, you will see that there are two print statements. The first print statement without the parentheses is a give-away that it will only work with Python 2:

print "Hello"  # No parentheses - This only works in Python 2, a dead give-away that this script is for Python 2.

print("Hello") # With parentheses - this will work in Python 2 and Python 3

Of course, you can always run the code against both Python 2 and 3 and see what happens.

We have now seen that there are two Python versions available by default on Raspbian OS, and made mention that there are system-level utilities that are written in Python that reply on these versions. As Python developers, we must take care not to disrupt the global Python installations as this can potentially break system-level utilities.

We will now turn our attention to a very important Python concept, the Python virtual environment, which is the way we isolate or sandbox our own Python projects from the global installation.

In this section, we will discuss how Python interacts with your operating system installation and cover the steps necessary to set up and configure a Python development environment. In addition, as part of our setup process, we will clone the GitHub repository that contains all of the code (organized by chapter) for this book.

By default, Python and its package management tool, pip, operate globally at the system level and can create some confusion for Python beginners because this global default is in contrast to many other language ecosystems that operate locally on a project folder level by default. Unwearyingly working and making changes to the global Python environment can break Python-based system-level tools, and remedying the situation can become a major headache.

As a Python developer, we use Python virtual environments to sandbox our Python projects so they will not adversely interfere with system-level Python utilities or other Python projects.

In this book, we will be using a virtual environment tool known as venv, which comes bundled as a built-in module with Python 3.3 and above. There are other virtual environment tools around, all with their relative strengths and weaknesses, but they all share the common goal of keeping Python dependencies isolated to a project.

Let's begin and clone the GitHub repository and create a new Python virtual environment for this chapter's source code. Open a new Terminal window and work through the following steps:

1. Change into or create a folder where you want to store this book's source code and execute the following commands. With the last command, we rename the cloned folder to be pyiot. This has been done to help shorten Terminal command examples throughout the book:

$ cd ~
$ git clone https://github.com/PacktPublishing/Practical-Python-Programming-for-IoT
$ mv Practical-Python-Programming-for-IoT pyiot

$ cd ~/pyiot/chapter01

3. Execute the following command, which creates a new Python virtual environment using the venv tool. It's important that you type python3 (with the 3) and remember that venv is only available with Python 3.3 and above:

$ python3 -m venv venv

The options that we are passing to python3 include -m venv, which tells the Python interpreter that we want to run the module named venv. The venv parameter is the name of the folder where your virtual environment will be created.

4. To use a Python virtual environment, we must activate it, which is accomplished with the activate command:

# From with in the folder ~/pyiot/chapter01
$ source venv/bin/activate
(venv) $

When your Terminal has a Python virtual environment activated, all Python-related activity is sandboxed to your virtual environment.

5. Next, execute which python (without the 3) in your Terminal, and notice that the location of the Python executable is beneath your venv folder and if you check the version of Python, it's Python version 3:

(venv) $ which python
/home/pi/pyiot/chapter01/venv/bin/python

(venv) $ python --version
Python 3.7.3

6. To leave an activated virtual environment, use the deactivate command as illustrated here:

(venv) $ deactivate
$ 

Notice also that (venv) $ is no longer part of the Terminal prompt text once the virtual environment has been deactivated.

7. Finally, now that you are outside of our Python virtual environment if you execute which python (without the 3) and python --version again, notice we're back to the default system-level Python interpreter, which is version 2:

$ which python
/usr/bin/python

$ python --version
Python 2.7.13

As we just illustrated in the preceding examples, when we ran python --version in an activated virtual environment, we see that it's Python version 3 whereas in the last example, at the start of this chapter, the system level, python --version, was version 2, and we needed to type python3 --version for version 3. In practice, python (with no number) relates to the default version of Python. Globally, this is version 2. In your virtual environment, we only have one version of Python, which is version 3, so it becomes the default.

We have now seen how to create, activate, and deactivate a Python virtual environment and why it is important to use a virtual environment to sandbox Python projects. This sandboxing means we can isolate our own Python projects and their library dependencies from one another, and it prevents us from potentially disrupting the system-level installation of Python and breaking any system-level tools and utilities that rely on them.

Next, we will see how to install and manage Python packages in a virtual environment using pip.

In this section, we learn how to install and manage Python packages in a Python virtual environment you created and explored in the previous section. A Python package (or library if you prefer that term) allows us to extend the core Python language with new features and functionality. 

We will need to install many different packages throughout this book, however, for starters and to explore and learn the basic concepts related to package installation and management, we will be installing two common GPIO-related packages in this section that we will use throughout this book. These two packages are the following:

• The GPIOZero library, an entry-level and easy to use GPIO library for controlling simple electronics
• The PiGPIO library, an advanced GPIO library with many features for more complex electronic interfacing

In the Python ecosystem, package management is done with the pip command (pip stands for Python installs packages). The official public package repository that pip queries is known as the Python Package Index, or simply PyPi, and it is available for browsing on the web at https://pypi.org.

There will be code examples in this book where we will be interacting with your Raspberry Pi's GPIO pins, so we need to install a Python package (or two) so that your Python code can work with your Raspberry Pi's GPIO pins. For now, we are just going to check for and install two GPIO-related packages. In Chapter 2, Getting Started with Python and IoT, and Chapter 5, Connecting Your Raspberry Pi to the Physical World, we will cover these GPIO packages and other alternatives in greater detail.

In your chapter01 source code folder, you will find a file named gpio_pkg_check.py, which is replicated in the following. We will use this file as the basis to learn about pip and package management in the context of a Python virtual environment. This script simply reports the availability of a Python package depending on whether using import succeeds or raises an exception:

"""
Source File: chapter01/gpio_pkg_check.py
"""
try:
    import gpiozero
    print('GPIOZero Available')
except:
    print('GPIOZero Unavailable. Install with "pip install gpiozero"')

try:
    import pigpio
    print('pigpio Available')
except:
    print('pigpio Unavailable. Install with "pip install pigpio"')

Let's check for the availability of GPIO packages using gpio_pkg_check.py and with pip. I'll kill the suspense by telling you that they're not going to be available in your freshly-created virtual environment (yet), however, we are going to install them!

The following steps will walk us through the process of upgrading pip, exploring the tool's options, and installing packages:

1. As the first step, we will upgrade the pip tool. In a Terminal window, run the following command, remembering that all commands that follow must be performed in an activated virtual environment—meaning you should see the text (venv) in the Terminal prompt:

(venv) $ pip install --upgrade pip
...output truncated...

The preceding upgrade command may take a minute or two complete and will potentially output a lot of text to the Terminal.

2. With pip now upgraded, we can see what Python packages are already installed in our virtual environment using the pip list command:

(venv) $ pip list
pip (9.0.1)
pkg-resources (0.0.0)
setuptools (33.1.1)

What we see in the preceding are the default Python packages in our fresh virtual environment. Do not worry if the exact package list or version numbers do not match exactly with the example.

3. Run our Python script with the python gpio_pkg_check.py command and observe that our GPIO packages are not installed:

(venv) $ python gpio_pkg_check.py
GPIOZero Unavailable. Install with "pip install gpiozero"
pigpio Unavailable. Install with "pip install pigpio"

4. To install our two required GPIO packages, we use the pip install command as shown in the following example:

(venv) $ pip install gpiozero pigpio
Collecting gpiozero...
... output truncated ...

5. Now, run the pip list command again; we will see these new packages are now installed in our virtual environment:

(venv) $ pip list
colorzero (1.1)
gpiozero (1.5.0)   # GPIOZero
pigpio (1.42)      # PiGPIO
pip (9.0.1)
pkg-resources (0.0.0)
setuptools (33.1.1)

You may have noticed that there is a package called colorzero (this is a color manipulation library) that we did not install. gpiozero (version 1.5.0) has a dependency on colorzero, so pip has installed it for us automatically.

6. Re-run python gpio_pkg_check.py and we now see that our Python modules are available for import:

(venv) $ python gpio_pkg_check.py
GPIOZero Available
pigpio Available

Great! We now have a virtual environment with two GPIO packages installed. As you work on Python projects, you will inevitably install more and more packages and want to keep track of them.

7. Take a snapshot of the packages you have previously installed with the pip freeze command:

(venv) $ pip freeze > requirements.txt

The preceding example freezes all installed packages into a file named requirements.txt, which is a common filename to use for this purpose.

8. Look inside the requirements.txt file and you will see all of the Python packages listed together with their version numbers:

(venv) $ cat requirements.txt
colorzero==1.1
gpiozero==1.5.0
pigpio==1.42
pkg-resources==0.0.0

In the future, if you move your Python project to another machine or a new virtual environment, you can use your requirement.txt file to install all of your captured packages in one go using the pip install -r requirements.txt command.

We've now completed the basic installation life cycle of Python packages using pip. Note that whenever you install new packages with pip install, you also need to re-run pip freeze > requirements.txt to capture the new packages and their dependencies.

To finish our exploration of pip and package management, here are a few other common pip commands:

# Remove a package
(venv) $ pip uninstall <package name>
 
# Search PyPi for a package (or point your web browser at https://pypi.org)
(venv) $ pip search <query text>
 
# See all pip commands and options (also see Further Reading at the end of the chapter).
(venv) $ pip --help

Congratulations! We've reached a milestone and covered the essential virtual environment principles that you can use for any Python project, even ones that are not Raspberry Pi related! 

Now that we have seen how to create a virtual environment and install packages, let's take a look at a typical Python project folder structure such as ~/pyiot/chapter01 and discover what lies beneath the venv folder.

This section relates to venv, which we have been using in this chapter, and will apply to virtualenv but not pipenv, which we listed as alternative virtual environment tools. The example is also specific to a Raspbian OS and is typical of a standard Unix-based OS. It's important to, at a minimum, understand the basic structure of a virtual environment deployment since we will be mixing our own Python programming code in with the files and folders that make up the virtual environment.

Here is the folder structure of our virtual environment. Yep, its a screenshot from a Mac. That's so I could get everything on screen at once:

The following points explain the core subfolders that are found within our ~/pyiot/chapter01 folder after we ran python3 -m venv venv and installed packages using pip:

• The venv folder contains all of the Python virtual environment files. There is no real practical need to be touching anything under this folder manually—let the tools do that for you. Remember that the folder is named venv only because that's what we called it when it was created.

• The venv/bin folder contains the Python interpreter (in the venv case, there are symbolic links to the system interpreter) and other core Python tools, including pip.
• Underneath the venv/lib folder are all the sandboxed Python packages for the virtual environment, including the GPIOZero and PiGPIO packages we installed using pip install.
• Our Python source file, gpio_pkg_check.py, is in the top-level folder, ~/pyiot/chapter01; however, you can create sub-folders here to help to organize your code and non-code files.
• Finally, requirements.txt lives by convention in the top project folder.

The virtual environment folder venv does not actually need to be kept in the project folder; however, it's often convenient to have it there for activation with the activate command.

It's important to understand that, as a Python developer, you will be mixing in your own programming code with files and folders that form part of the virtual environment system and that you should be pragmatic when selecting which files and folders are added to your version control system, should you be using one.

This last point is important since the virtual environment system can amount to many megabytes in size (and often many times larger than your program code) that does not need versioning (since we can always recreate the virtual environment as long as we have a requirements.txt file), plus it's host platform-specific (that is, there will be differences between Windows, Mac, and Linux), plus there will be differences between different virtual environment tools (for example, venv versus pipenv). As such, virtual environments are not generally portable in projects that involve many developers working on different computers.

Now that we have briefly explored the file and folders structure and the importance of understanding this structure, we will continue and look at alternative ways of running a script that is sandboxed to a virtual environment.

Let's briefly turn our attention to the alternative ways that we can execute a Python script. As we will learn, choosing the appropriate method is all based around how and from where you intend to start your script and whether your code requires elevated permissions.

The most common way of running a Python script is from within its virtual environment and with the permissions of the currently logged in user. However, there will be scenarios where we need to run a script as the root user or from outside an activated virtual environment.

Here are the ways we will explore:

• Using sudo with virtual environments
• Executing Python scripts outside of their virtual environments
• Running a Python script at boot

Let's start by learning how to run a Python script with root user permissions.

I'm sure that while working on your Raspberry Pi you have had to execute commands in a Terminal with the sudo prefix because they required root privileges. If you ever need to run a Python script that is in a virtual environment as root, you must use the full path to your virtual environment's Python interpreter.

Simply prefixing sudo before python, as shown in the following example, does not work under most circumstances, even if we are in the virtual environment. The sudo action will use the default Python that's available to the root user, as shown in the second half of the example:

# Won't work as you might expect!
(venv) $ sudo python my_script.py
 
# Here is what the root user uses as 'python' (which is actually Python version 2).
(venv) $ sudo which python
/usr/bin/python

The correct way to run a script as root is to pass the absolute path to your virtual environment's Python interpreter. We can find the absolute path using the which python command from inside an activated virtual environment:

(venv) $ which python
/home/pi/pyiot/chapter01/venv/bin/python

Now, we sudo our virtual environment's Python interpreter and the script will run as the root user and within the content of our virtual environment:

(venv) $ sudo /home/pi/pyiot/chapter01/venv/bin/python my_script.py

Next, we'll see how to run a Python script that's sandboxed in a virtual environment from outside of its virtual environment. 

A natural extension to the preceding discussion on sudo is how do I run a Python script from outside of its virtual environment? The answer is the same as in the preceding section: just make sure you are using the absolute path to your virtual environment's Python interpreter.

The following command will run a script as the currently logged in user (which, by default, is the pi user):

# Run script as logged-in user.
$ /home/pi/pyiot/chapter01/venv/bin/python gpio_pkg_check.py

Or to run the script as root, prefix sudo:

# Run script as root user by prefixing sudo
$ sudo /home/pi/pyiot/chapter01/venv/bin/python gpio_pkg_check.py

Since we are using the virtual environment's Python interpreter, we are still sandboxed to our virtual environment and any Python packages we installed are available.

Next, we will learn how to make a Python script run whenever you boot your Raspberry Pi.

There will come a time when you have developed an awesome IoT project and you want it to run automatically every time you start your Raspberry Pi. Here is one simple way to achieve this using a feature of cron, the Unix scheduler. If you are not familiar with the basics of cron, search the web for cron tutorial—you'll find heaps of them. I've provided curated links in the Further reading section.

Here are the steps to configure cron and make a script run on boot:

1. In your project folder, create a bash script. I've named it run_on_boot.sh:

#!/bin/bash

# Absolute path to virtual environment python interpreter
PYTHON=/home/pi/pyiot/chapter01/venv/bin/python

# Absolute path to Python script
SCRIPT=/home/pi/pyiot/chapter01/gpio_pkg_check.py

# Absolute path to output log file
LOG=/home/pi/pyiot/chapter01/gpio_pkg_check.log

echo -e "\n####### STARTUP $(date) ######\n" >> $LOG
$PYTHON $SCRIPT >> $LOG 2>&1

This bash script will run a Python script using the absolute paths for both the script and its Python interpreter. Also, it captures any script output and stores it in a log file. For this example, we're simply going to run and log the output of gpio_pkg_check.py on boot. It's the last line that ties everything together and runs and logs our Python script. The 2>&1 part at the end is necessary to ensure that errors, in addition to standard output, are also logged.

2. Mark the run_on_boot.sh file as an executable file:

$ chmod u+x run_on_boot.sh

If you are not familiar with the chmod command (chmod means change mode), what we are doing is giving the operating system permission to execute the run_on_boot.sh file. The u+x parameters mean for the current User, make the file eXecutable. To learn more about chmod, you can type chmod --help or man chmod in the Terminal.

3. Edit your crontab file, which is the file where cron scheduling rules are stored:

$ crontab -e

4. Add the following entry to your crontab file, using the absolute path to the run_on_boot.sh bash script we created in step 1:

@reboot /home/pi/pyiot/chapter01/run_on_boot.sh &

Do not forget the & character at the end of the line. This makes sure the script runs in the background.

5. Run the run_on_boot.sh file manually in a Terminal to make sure it works. The gpio_pkg_check.log file should be created and contains the output of the Python script:

$ ./run_on_boot.sh
$ cat gpio_pkg_check.log
####### STARTUP Fri 13 Sep 2019 03:59:58 PM AEST ######
GPIOZero Available
PiGPIO Available

6. Reboot your Raspberry Pi:

$ sudo reboot

7. Once your Raspberry Pi has finished restarting, the gpio_pkg_check.log file should now contain additional lines, indicating that the script did indeed run at boot:

$ cd ~/pyiot/chapter01
$ cat gpio_pkg_check.log

####### STARTUP Fri 13 Sep 2019 03:59:58 PM AEST ######

GPIOZero Available
PiGPIO Available

####### STARTUP Fri 13 Sep 2019 04:06:12 PM AEST ######

GPIOZero Available
PiGPIO Available

If you are not seeing the additional output in the gpio_pkg_check.log file after a reboot, double-check that the absolute path you entered in crontab is correct and that it works manually as per step 5. Also, review the system log file,  /var/log/syslog, and search for the text, run_on_boot.sh.

We have now learned alternative methods to run a Python script, which will help you in the future to correctly run your Python-based IoT projects after they are developed or start them when your Raspberry Pi boots if required.

Next, we will now move on to making sure your Raspberry Pi is set up and configured correctly for the GPIO and electronic interfacing that we'll be diving into in the next chapter, Chapter 2, Getting Started with Python and IoT, and subsequent chapters.

Before we can start working with Python GPIO libraries and controlling electronics, one task we need to perform is to enable the GPIO interfaces on your Raspberry Pi. Even though we have installed Python packages for GPIO control, we have not told Raspbian OS that we want to use the Raspberry Pi's GPIO Pins for specific cases. Let's do that now.

Here are the steps to follow:

1. From your Raspbian desktop, navigate to the Raspberry menu | Preferences | Raspberry Pi Configuration, as shown here in Figure 1.2:

2. Enable all of the interfaces as shown in the following screenshot:

3. Click the OK button.

After you click the OK button, you may be prompted to reboot your Raspberry Pi; however, do not confirm the reboot just yet because there is one more task we need to perform first. We'll look at that next.

We also need to start the PiGPIO daemon, which is a system service that needs to be running so that we can use the PiGPIO GPIO client library, which we will start using next in Chapter 2, Getting Started with Python and IoT.

Execute the following in a Terminal. This will start the PiGPIO daemon and will ensure that the PiGPIO daemon is started automatically when your Raspberry Pi boots:

$ sudo systemctl enable pigpiod
$ sudo systemctl start pigpiod

Now, it's time to reboot your Raspberry Pi! So, take a break while your Raspberry Pi restarts. You deserve it because we have covered a lot! 

In this chapter, we explored the Python ecosystem that is part of a typical Unix-based operating system such as Raspbian OS and learned that Python is a core element of the operating system tooling. We then covered how to create and navigate a Python virtual environment so that we can sandbox our Python projects so they will not interfere with one another or the system-level Python ecosystem.

Next, we learned how to use the Python package management tool, pip, to install and manage Python library dependencies from within a virtual environment, and we did this by installing the GPIOZero and PiGPIO libraries. And since there will be times that we need to execute a Python script as the root user, from outside its virtual environment or during boot up, we also covered these various techniques.

By default, Raspbian does not have all of its GPIO interfaces enabled, so we performed the configuration needed to enable these features so that they are readily available for use in later chapters. We also started and learned how to set up the PiGPIO daemon service so that it starts every time your Raspberry Pi is booted.

The core knowledge you have gained in this chapter will help you to correctly set up and navigate sandboxed Python development environments for your own IoT (and non-IoT) projects and safely install library dependencies so they do not interfere with your other Python projects or the system-level installation of Python. Your understanding of different ways of executing a Python program will also help you to run your projects with elevated user permissions (that is, as the root user), or at boot, should your project have these requirements.

Next, in Chapter 2, Getting Started with Python and IoT, we will jump straight into Python and electronics and create an end-to-end internet-enabled program that can control an LED over the internet. We will take a look at two alternative ways of flashing an LED using the GPIOZero and PiGPIO GPIO libraries before connecting our LED to the internet by using an online service, dweet.io, as our networking layer. 

As we conclude, here is a list of questions for you to test your knowledge regarding this chapter's material. You will find the answers in the Assessments section of the book:

1. What is the main reason why you should always use a virtual environment for your Python projects?
2. Do you need to or should you place the virtual environment folder (that is, venv) under version control?
3. Why create a requirements.txt file?
4. You need to run a Python script as the root user. What step must you take to ensure that the script executes in its intended virtual environment context?
5. What does the source venv/bin/activate command do?
6. You are in an activated virtual environment. What is the command to leave the virtual environment and return to the host shell?
7. You created a Python project and virtual environment in PyCharm. Can you work on and run the project's Python scripts in a Terminal?
8. You want a GUI tool to edit and test Python code on your Raspberry Pi but do not have PyCharm installed. What pre-installed tool that comes with Python and Raspbian could you use?
9. You've advanced in your Python and electronics knowledge and are trying to hook up a device using I2C to your Raspberry Pi but you cannot get it to work. What might be the problem and how do you address it?

We covered the venv virtual environment tool in this chapter. Here are links to its official documentation:

• venv documentation: https://docs.python.org/3/library/venv.html
• venv tutorial: https://docs.python.org/3/tutorial/venv.html

If you would like to learn about the virtualenv and pipenv alternative virtual environment tools, here is their official documentation:

• virtualenv home page: https://virtualenv.pypa.io/en/latest
• pipenv home page: https://docs.pipenv.org/en/latest

The following is a link to the Python Packaging Guide. Here you will find a comprehensive guide regarding Python package management, including pip and the easy-install/setup tools alternatives: 

• Python Packaging User Guide: https://packaging.python.org

If you wish to learn more about scheduling and cron, here are two resources to get you started:

• An overview of cron syntax (and a GUI tool): https://www.raspberrypi.org/documentation/linux/usage/cron.md 
• A detailed tutorial on cron syntax: https://opensource.com/article/17/11/how-use-cron-linux