Chapter 13: Robot Vision – Using a Pi Camera and OpenCV

Giving a robot the ability to see things allows it to behave in ways to which humans relate well. Computer vision is still actively being researched, but some of the basics are already available for use in our code, with a Pi Camera and a little work.

In this chapter, we will use the robot and camera to drive to objects and follow faces with our pan-and-tilt mechanism. We'll be using the PID algorithm some more and streaming camera output to a web page, giving you a way to see what your robot is seeing.

The following topics will be covered in this chapter:

• Setting up the Raspberry Pi camera
• Setting up computer vision software
• Building a Raspberry Pi camera stream app
• Running background tasks when streaming
• Following colored objects with Python
• Tracking faces with Python

Technical requirements

For this chapter, you will need the following:

• The robot with the pan-and-tilt mechanism from Chapter 11, Programming Encoders with Python.
• Code for the robot up to Chapter 11, Programming Encoders with Python, which you can download from GitHub at https://github.com/PacktPublishing/Learn-Robotics-Programming-Second-Edition/tree/master/chapter11. We will be extending and modifying this for new functionality.
• A Raspberry Pi camera.
• A 300 mm-long Pi Camera cable, as the cable included with the camera is too short. Be sure that the cable is not for a Pi Zero (which has different connectors).
• Two M2 bolts and an M2 nut.
• A small square of thin cardboard—a cereal box will do.
• A small jeweler's screwdriver.
• A pencil.
• A kids' bowling set—the type with differently colored pins (plain, with no pictures).
• A well-lit space for the robot to drive in.
• Internet access.

The code for this chapter is...

Setting up the Raspberry Pi camera

Before we can get into computer vision, we need to prepare the camera on your robot. There is hardware installation and software installation involved.

When we have completed this installation, our robot block diagram will look like Figure 13.1:

Figure 13.1 – Our robot block diagram with the camera added

Figure 13.1 continues the block diagrams we have shown throughout the book, with the camera's addition and its connection to the Raspberry Pi highlighted on the left.

We will first attach the camera to the pan-and-tilt assembly. We can then use a longer cable to wire the camera into the Pi. Let's start preparing the camera to be attached.

Attaching the camera to the pan-and-tilt mechanism

In Chapter 10, Using Python to Control Servo Motors, you added a pan-and-tilt mechanism to your robot. You will mount the camera onto the front plate of this mechanism. There are brackets and kits, but they...

Setting up computer vision software

Before we can start writing code, we'll need to set up drivers, tools, and libraries to interact with the camera and software to assist with computer vision.

In this section, we will activate the camera in Raspberry Pi OS Raspberry Pi OS and get a test picture. Then we will add the libraries to start interacting with the camera for visual processing.

We will then build our first app with the tool to demonstrate that the parts are in place and give us a starting point for the behaviors. Let's get into setting up the software.

Setting up the Pi Camera software

So that the camera is ready to use, we need to enable it:

1. Power up the Pi on external power (that is, plugged into a USB wall adapter) for this operation, leaving the motors powered down for now.
2. Log in via SSH. At the terminal, type the following:

   pi@myrobot:~ $ sudo raspi-config

3. You should now see raspi-config. Select the Interfacing Options menu item by...

Building a Raspberry Pi camera stream app

Downloading one picture at a time is fine, but we need to do things with those pictures on our robot. We also need a handy way to see what the robot is doing with the camera data. For that, we will learn how to use a Flask web server to serve up our pictures so we can view the output on a phone or laptop. We can use the core of this app to make a few different behaviors. We'll keep the base app around for them.

A video or video stream is a sequence of images, usually known as frames.

Let's design our streaming server.

Designing the OpenCV camera server

The diagram in Figure 13.9 shows an image data pipeline, going from the camera, through the processing, and out to our web browser:

Figure 13.9 – The image server app

The image server app in Figure 13.9 starts with the camera. The camera feeds image data to a convert to OpenCV step, with the raw photo given. Image data needs some processing...

Running background tasks when streaming

Our image service works but has a significant flaw. Currently it will wait between requests before taking each action, but what if we want our robot to be doing something? To do this, we need to be able to run a behavior in parallel with the server. That behavior and the server both need access to the image data.

We will approach this by making the Flask web app a secondary process, with the behavior as the primary process for the robot when it is running. Python has a handy tool for precisely this kind of structure, called multiprocessing. Find out more at https://docs.python.org/3/library/multiprocessing.html.

Communicating between multiple processes is tricky. If two processes try to access (read or write) the same data simultaneously, the results can be unpredictable and cause strange behavior. So, to save them trying to access data simultaneously, we will use the multiprocessing queue object. A queue allows one process to put data...

Following colored objects with Python

Now we have some basics ready; we can use this to build some more interesting behaviors.

We will create a behavior that will chase, but not get too close to, a colored object. This behavior will make the robot seem very intelligent. We will revisit color models, covered in Chapter 9, Programming RGB Strips in Python. We'll add color masking and filtering and use the OpenCV contours tools to detect the largest blob of color in an image and point the robot at it.

Building the color-chasing behavior requires a few steps. Let's start with a diagram showing an overview of this whole behavior in Figure 13.12:

Figure 13.12 – The color-tracking behavior

The flow of data in Figure 13.12 starts from camera images. These go through visual processing to get object info from image. get object info from image outputs the object's size (based on the radius of a circle around it) and the object's...

Tracking faces with Python

Detecting faces (or other objects) by features is a smart behavior. Once our robot is detecting faces, it will point the pan-and-tilt mechanism at the nearest (well, largest) face.

Using Haar cascades is a common technique, well documented in a paper by Paul Viola and Michael Jones (known as Viola Jones). In essence, it means using a cascade of feature matches to search for a matching object. We will give an overview of this technique, then put it into use on our robot to create a fun behavior. Using different cascade model files, we could pick out faces or other objects.

Finding objects in an image

We will be using an algorithm implemented in OpenCV as a single and useful function, which makes it very easy to use. It provides a simple way to detect objects. More advanced and complex methods involve machine learning, but many systems use Haar cascades, including camera apps on phones. Our code will convert the images into grayscale (black through...

Summary

In this chapter, you saw how to set up the Raspberry Pi Camera module. You then used it to see what your robot sees—the robot's view of the world.

You got the robot to display its camera as a web app on a phone or desktop, and then used the camera to drive smart color- and face-tracking behaviors. I've suggested ways the behaviors could be enhanced and hopefully given you a taste of what computer vision can do.

In the next chapter, we will extend our object-tracking visual processing to follow lines with the camera, seeing further ways to use the camera.

Exercises

This code is fun, but there are many ways you could improve the behaviors. Here are some suggested ways to extend this code and deepen your learning:

• Use the control pipeline to allow a user to tune the color filters, correct radius, and PID values from the web page. Perhaps the initial PID values should be close to the other tunable values?
• There is quite a lot of setup code. Could you put this into a function/method?
• Could the queues to the web page be used to send the debug data to the page, instead of printing them in the console? Could the data be plotted in a graph?
• The field of view for tracking with the Pi Camera is pretty narrow. A wide-angle lens would improve the field of view a lot, letting the robot see more.
• The camera doesn't perform too well when it's dark. The robot has an LED strip, but it's not illuminating much. Could you add a bright LED as a headlamp for the camera?
• You could track other objects by trying the...

Further reading

Visual processing is a deep topic, so this is only a small selection of places where you can read more about using a camera for visual processing:

• The Official Raspberry Pi Camera Guide at https://magpi.raspberrypi.org/books/camera-guide is an excellent resource for getting to know the camera, with many practical projects for it.
• To delve in far greater depth into using the Raspberry Pi Camera, I recommend the PiCamera documentation, available at https://picamera.readthedocs.io/.
• To gain insight into further techniques, the PyImageSearch website, at https://www.pyimagesearch.com, has great resources.
• OpenCV and visual processing is a complex topic, only briefly covered here. I recommend OpenCV 3 Computer Vision with Python Cookbook, by Alexey Spizhevoy and Aleksandr Rybnikov, from Packt Publishing, available at https://www.packtpub.com/application-development/opencv-3-computer-vision-python-cookbook, for more information.
• Streaming video through...