Chapter 7. Creating an Image Sensor Using a Camera and OpenCV

There has been an enormous amount of research behind a small name—image sensor. From developing CMOS sensors, which can take crystal-clear images, to employing enhancement of images using image processing, there are so many efforts involved. Making different types of cameras suitable for different applications and processing algorithms is not easy at all.

Image sensors are used in cameras, and they are very difficult to use in unpackaged conditions, as they are fragile and sensitive to electrostatic discharge. We will be using a camera (basically a lens with the protective case and an interface cable) to demonstrate image capturing using a RasPi. Performance of a camera with a RasPi is always rated low in discussions because of its poor performance with image processing and video processing, with comments that image processing and video capturing is not the RasPi's strong point. Somehow, these comments are true when we run computation...

Image processing

Have you ever tried to look at an image by zooming it to a maximum level? It just looks like a floor with organized tiles and colorful patterns on it. These square tiles in the image are known as pixels. Basically, an image is a group of such pixels, with each pixel containing a particular value of color, which forms the recognizable patterns by providing information to the human eye. It all depends on how humans perceive the image by observing shapes and colors. Each pixel in an image contains information that can be generated from a byte (8 bits) or a couple of bytes, which defines the depth of an image. Depth of an image is nothing but the number of bits present in a single pixel. Current display monitors and graphics engines support up to 64-bit depth of images. Basic types of images are binary, grayscale, and RGB, and many more such as HSV, HLS, and YCC are known types. A grayscale image does have the range of values from 0 to 255 in the 8-bit mode, while a binary image...

OpenCV

One of the most powerful and widely used cross-platform libraries for image processing is OpenCV, which can run on any hardware or software platform. It can be installed on the RasPi to get functionality such as face detection, finding contours, gesture recognition, motion tracking, 3D depth perception using two cameras, and many more complex algorithms for machine learning. OpenCV stands for Open Source Computer Vision. It was developed by Intel. It is basically written in the C++ language and can have full interfacing with Python, Java, or MATLAB. Check out http://opensource.org for the list of available open source libraries. With aiming for computational efficiency in mind, OpenCV is designed to work with multicore architectures. Writing OpenCV in C and C++ is common across the globe among image processing experts. OpenCV can also be written in the Python language, as the library has full development and supports functions in the Python script.

Computer vision is a quickly growing...

Camera interfacing with the RasPi

The RasPi has the Broadcom multimedia processor BCM2835, which has an integrated Videocore-IV graphics engine. There are multiple ways to interface the camera with the RasPi connectors. As described in Chapter 1, Meeting Your Buddy – the Raspberry Pi, (refer to the diagram showing the RasPi's port description), the RasPi has a dedicated CSI port for connecting the camera module. This port can be located near the Ethernet controller (in the case of model B) or near the 4-pole audio connector (in the case of model B+). The Camera Serial Interface (CSI) port is dedicated to camera interfacing in mobile devices and advanced peripheral controllers. We have two options to interface the cameras with our RasPi. We can either interface the webcam via USB, or choose the camera module from the Raspberry Pi organization known as the Raspberry Pi camera module.

The RasPi camera modules

There are two options when buying the dedicated camera modules to be interfaced with...

Live streaming using a network camera

Have you ever wondered whether the RasPi can be used for live streaming purposes? We are going to stream videos on the RasPi on the localhost network using RasPi as a network camera. There are multiple ways to make use of the RasPi for live streaming:

• By fixing a camera interfaced with the RasPi being in the same network, you can SSH through the connection and open the camera port through the command-line interface

• Use open source software motion to enable live streaming at the local web browser

We will be using motion to get access to the RasPi camera. Motion has a built-in HTTP server that enables the camera view from the web browser. Motion can record videos in MPEG format and capture images in JPEG format. These images and videos can be stored wherever you want, and it does support databases such as MySQL. Enter these commands to install motion on the RasPi:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -y install bison
sudo apt-get -y install...

Porting OpenCV

Porting OpenCV on RasPi is the lengthiest task compared to installing other libraries. Installing the suitable and companion libraries is necessary for effective running of the image processing algorithms. It has been observed in the community that not many have succeeded in porting the OpenCV properly. We will go through a step-by-step guide that will give you an up-and-running OpenCV at the first time. While installing some of the library, the order of the installation doesn't matter, but it is suggested that the flow given here will give you the exact results you are looking for.

A quick checklist is as follows:

• An Internet connection shared with RasPi through a Wi-Fi adapter or a PC.

• Use PuTTY to log in to the RasPi using an Ethernet connection from a PC.

• Enough space (up to 4 GB) on the RasPi SD card. Use the df -h command to check out the free space on the SD Card.

• Interface the camera when needed. A USB camera needs enough current to operate. Connect a 2A adapter to the...

Create a motion detector

We have already set up a piece of software called motion. It detects and captures motion, but that's not fun at all compared to building our own code to capture motion. We will create code that will detect motion and take a decision according to the extent of motion. We will trigger an LED to indicate the decision taken by the RasPi. This code is going to be a bit long, so the explanation is given in comments within the code. However, important information is provided just after the end of the code:

#include <iostream>
#include <cv.h>
#include <highgui.h>
#include <stdio.h>
#include <cctype>
#include <iterator>
#include <unistd.h>
#include <wiringPi.h>
#include "opencv2/highgui/highgui.hpp"
#include <time.h>

using namespace cv;
using namespace std;

int main()
{
/*----Section 1 -----------------Declarations------------------*/
wiringPiSetup();  //To toggle LED, Essential
pinMode(0, OUTPUT); //BCM_GPIO pin 17
CvMemStorage...

Amazing projects for you

With some knowledge on OpenCV, you can go ahead and read the manuals of OpenCV to gain more knowledge on how functions and arguments are passed. Initially, it can be somewhat difficult for you to start OpenCV using image pointers. There is an immense amount of help available at http://www.opencv.org. While taking help from the Internet and the code available from the official website, check for the OpenCV language (platform) first. It has been observed that when C code merges with C++ code, it doesn't work even after banging your head for hours. It is advised to keep The OpenCV Reference Manual (Current Release 2.4.9.0) with you while coding OpenCV.

It would be quite a good challenge for you to develop a project that fulfils the following requirements:

• Capture live images using the interfaced camera, split the image into segments of the same size (say, four vertical segments), and compare it with the reference image. There will be a clear difference in one of the segments...

Summary

This chapter was totally dedicated to development in image processing using the most advanced OpenCV library. We started with an overview of image processing and its applications, followed by an introduction to OpenCV. By now, you already know that the OpenCV library is so vast that even four books like this would not be enough to provide a detailed description of the entire library. You then understood the camera interface on the RasPi module, and we chose the USB camera to go with. Next, you experimented live streaming using the RasPi as a network camera module (IP camera) to keep track of your backyard directly from the lounge.

We started porting the OpenCV library, which took almost half a day to be up and running. Then, we verified the library using the camera interface. Next, we started building the project in the C language to detect human motion by calculating a motion-affected area. A useful shell file was prepared to compile the OpenCV code along with the wiringPi interface...