Chapter 5. Automated Optical Inspection, Object Segmentation, and Detection

In the previous chapter, we learned about histograms and filters that allowed us to understand image manipulation and create a photo application.

In this chapter, we will introduce you to the basic concepts of object segmentation and detection, which means isolation the objects that appear in an image for future processing and analysis.

In this chapter, we will cover the following topics:

• Noise removal

• The basics of light/background removal

• The thresholding operation

• A connected component for object segmentation

• Finding contours for object segmentation

The industry sector uses complex Computer Vision systems and hardware. Computer Vision tries to detect the problems and minimizes errors produced in the production process and increases the quality of final products.

In this sector, the name for Computer Vision tasks is Automated Optical Inspection or AOI. This name appears in the inspection of printed circuit board manufacturers...

Isolating objects in a scene

In this section, we will introduce you to the first step of any AOI algorithm, that is, isolating different parts or objects in a scene.

We will take the example of object detection and classification of three object types: a screw, a packing ring, and a nut and develop these in this chapter and Chapter 6, Learning Object Classification.

Let's say we are in a company that produces these three objects. All of them are in the same carrier tape, and our objective is to detect each object in the carrier tape and classify each one to allow a robot to put each object on the correct shelf:

In this chapter, we will isolate each object and detect its position in the image in pixels. In the next chapter, we will classify each isolated object to check whether it is a nut, a screw, or a packing ring.

In the following image, we show our desired result where there are a few objects in the left-hand side image, and in the right-hand side image, we draw each one in different colors...

Creating an application for AOI

To create our new application, we require a few input parameters when the user executes them; all of them are optional, excluding the input image to be processed:

• An input image to be processed

• The light image pattern

• The light operation, where the user can choose between difference or division operations:

  • If the input value of the user is set to 0, then a difference operation is applied

  • If the input value of the user is set to 1, then a division operation is applied

• Segmentation, where the user can choose between connected components with or without statistics and findContours methods:

  • If the input value of the user is set to 1, then the connected components method for the segment is applied

  • If the input value of the user is set to 2, then the connected components with the statistics area is applied

  • If the input value of the user is set to 3, then the findContours method is applied to the segmentation

To enable this user selection, we will use the command line parser...

Preprocessing the input image

This section introduces you to some of the most common techniques that can be applied to preprocess images in the context of object segmentation/detection. The preprocess is the first change that we make in a new image before we start with our work and extract the information that we require from it.

Normally, in the preprocessing step, we try to minimize the image noise, light conditions, or image deformations due to the camera lens. These steps minimize the errors when you try to detect objects or segment our image.

Noise removal

If we don't remove the noise, we can detect more objects than we expect because normally noise is represented as a small point in the image and can be segmented as an object. The sensor and scanner circuit normally produce this noise. This variation of brightness or color can be represented in different types, such as Gaussian noise, spike noise, and shot noise. There are different techniques that can be used to remove the noise. We...

Segmenting our input image

Now, we will introduce you to the following two techniques used to segment our thresholded image:

• The connected components

• The findContours function

With these two techniques, we will be allowed to extract each region of interest of our image where our target objects appear; in our case, a nut, screw, and ring.

The connected component algorithm

The connected component is a very common algorithm used to segment and identify parts in binary images. A connected component is an iterative algorithm used for the purpose of labeling an image using an 8- or 4-connectivity pixel. Two pixels are connected if they have the same value and are neighbors. In the following figure, each pixel has eight neighbor pixels:

A 4-connectivity means that only the 2, 4, 5, and 7 neighbors can be connected to the center if they have the same value. In the case of 8-connectivity, 1, 2, 3, 4, 5, 6, 7, and 8 can be connected if they have the same value.

In the following example, we can see the difference...

Summary

In this chapter, we explored the basics of object segmentation in a controlled situation, where a camera take pictures of different objects.

We learned how to remove the background and light in order to allow us to binarize our image by minimizing the noise and also three different algorithms used to divide and separate each object of an image, allowing us to isolate each object in order to manipulate or extract features. Finally, we extracted all the objects on an image, where we are going to extract characteristics of each of these objects to train a machine learning system.

In the next chapter, we are going to predict the class of any of objects in an image, and then call to a robot or any other system to pick any of them, or detect an object that is not in the correct carrier tape, and then notify to a person to pick it up.