Finding Road Markings Using OpenCV

In this chapter, we will use the concepts from Chapter 4, Computer Vision for Self-Driving Cars, to design a pipeline to identify road markings for self-driving cars using the OpenCV library. In general, we will preprocess our data with the OpenCV library and then feed in a deep learning network.

The main purpose of this chapter is to build a program that can identify road markings in a picture or a video. When we drive a car, we can see where the road markings are. A car obviously doesn't have any eyes, which is where computer vision comes in. In this chapter, we will use a complex algorithm that helps the computer see the world as we do. We will be using a series of camera images to identify road markings.

In this chapter, we will cover the following topics:

• Finding road markings in an image
• Detecting road markings in a video

Finding road markings in an image

Finding road markings is the first step toward building a self-driving car. If a camera sensor can detect road markings correctly, it can follow the lanes and drive safely. 

 The main steps for detecting road markings are as follows:

1. Loading the image using OpenCV
2. Converting the image into grayscale
3. Smoothing the image
4. Canny edge detection

5. Masking the region of interest in an image
6. Applying bitwise_and
7. Implementing the Hough transform
8. Optimizing the detected road markings
9. Detecting road markings in an image
10. Detecting road masking in a video

We will look at these steps in-depth in the following sections.

Loading the image using OpenCV

The first step involves is importing the image using OpenCV:

1. We will load the image using the imread function and display it using the imshow function. Let's import the libraries and load the image:

In[1]: import cv2
In[2]: image = cv2.imread('test_image.jpg')
In[3]: cv2.imshow('input_image', image)
In[4]: cv2.waitKey(0)
In[5]: cv2.destroyAllWindows()

2. The imported image looks as follows: 

Fig 5.1: The input image

In the next section, we will convert the RGB image into a grayscale image.

Converting the image into grayscale

We learned in Chapter 4, Computer Vision for Self-Driving Cars, that a three-channel color image has red, green, and blue channels (each pixel being a combination of these three channel values). A grayscale image has only one channel for each pixel (with 0 being black and 255 being white). Naturally, processing a single-channel image is faster than processing a three-channel color image, and it is less computationally expensive, too.

Also, in this chapter, we will develop an edge-detection algorithm. The edge-detection algorithm's main goal is to identify the boundaries of the objects within an image. Later in this chapter, we will be detecting edges to find a region in an image with a sharp change in the pixels. 

Now, as a first step, we will convert the image into grayscale:

1. Import the following libraries, which we need to convert the image into grayscale:

In[1]: import cv2
In[2]: import numpy as np

2. Read in and...

Smoothing the image

In this section, we will smooth the image with a Gaussian filter. Applying the GaussianBlur function from the OpenCV library reduces the noise in our image.

We will apply it using OpenCV, as follows:

1. Start by importing OpenCV and numpy:

In[1]: import cv2
In[2]: import numpy as np

2. In the next step, read the input image:

In[3]: image = cv2.imread('test_image.jpg')
In[4]: lanelines_image = np.copy(image)

3. Now, we will convert the image into grayscale:

In[5]: gray_conversion= cv2.cvtColor(lanelines_image, cv2.COLOR_RGB2GRAY)

4. Apply GaussianBlur using the OpenCV library:

In[6]: blur_conversion = cv2.GaussianBlur(gray_conversion, (5,5),0)
In[7]: cv2.imshow('input_image', blur_conversion)
In[8]: cv2.waitKey(0)
In[9]: cv2.destroyAllWindows()

The output is as follows:

Fig 5.3: Image after applying Gaussian blur

We smoothed the image and removed noise from it. In the next section, we will perform canny edge detection on the input image.

...

Canny edge detection

We covered canny edge detection in Chapter 4, Computer Vision for Self-Driving Cars. In this section, we will apply canny edge detection to identify the edges in the image. As we learned in Chapter 4, Computer Vision for Self-Driving Cars, an edge is a region in an image where there is a sharp change in intensity or a sharp change in color between adjacent pixels in an image. This change over a series of pixels is known as the gradient. As we already know, the canny function computes the gradient in all directions of a blurred image and will trace the strongest gradient as a series of pixels.

Now, we will implement canny edge detection using the OpenCV library:

1. Import the libraries we need:

In[1]: import cv2
In[2]: import numpy as np

2. Now, apply canny edge detection using OpenCV—specifically, using cv2.Canny:

In[3]: image = cv2.imread('test_image.jpg')
In[4]: lanelines_image = np.copy(image)
In[5]: gray_conversion= cv2.cvtColor(lanelines_image...

Masking the region of interest

We learned about region-of-interest masking in Chapter 4, Computer Vision for Self-Driving Cars. The next step toward identifying road markings is to mask the region of interest in our image:

1. Import the required libraries:

In[1]: import cv2
In[2]: import numpy as np
In[3]: import matplotlib.pyplot as plt

2. Next, write a function for canny edge detection:

In[4]: def canny_edge(image):
          gray_conversion= cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
          blur_conversion = cv2.GaussianBlur(gray_conversion, (5,5),0)
          canny_conversion = cv2.Canny(blur_conversion, 50,150)
          return canny_conversion

3. Now, we will write a function for the region-of-interest masking. We will check the image manually to identify the inputs as polygons. We will plot the image and then verify the points mentioned in the polygons:

In[5]: def reg_of_interest(image):
        Image_height = image.shape[0]
        polygons = np.array([[(200, Image_height), (1100...

Applying bitwise_and

In this section, we will apply bitwise_and and multiply all the bits in the black region of the image by 0000 and the white region by 1111, as shown in the following screenshot:

Fig 5.7: bitwise_and used on the black and white images

The bitwise_and conversion is as follows:

Fig 5.8: The bitwise_and conversion

Now, we will implement bitwise_and using OpenCV:

1. First, import the required libraries:

In[1]: import cv2
In[2]: import numpy as np
In[3]: import matplotlib.pyplot as plt

2. Then, write a canny edge detection function:

In[4]: def canny_edge(image):
         gray_conversion= cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
         blur_conversion = cv2.GaussianBlur(gray_conversion, (5,5),0)
         canny_conversion = cv2.Canny(blur_conversion, 50,150)
         return canny_conversion

3. Modify the region-of-interest masking function by adding bitwise_and:

In[5]: def reg_of_interest(image):
         image_height = image.shape[0]
         polygons...

Applying the Hough transform

In Chapter 4, Computer Vision for Self-Driving Cars, we looked at the theory behind the Hough transform. We also saw the differences between points in image space and Hough space, as shown in the following screenshot:

Fig 5.10: Image space to Hough space

Now, we will implement the Hough transform using OpenCV:

1. Import the required libraries:

In[1]: import cv2
In[2]: import numpy as np
In[3]: import matplotlib.pyplot as plt

2. Use the canny_edge function to detect of edges in the image:

In[4]: def canny_egde(image):
         gray_conversion= cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
         blur_conversion = cv2.GaussianBlur(gray_conversion, (5,5),0)
         canny_conversion = cv2.Canny(blur_conversion, 50,150)
         return canny_conversion

3. We will reuse the region-of-interest function from the previous section:

In[5]: def reg_of_interest(image):
         image_height = image.shape[0]
         polygons = np.array([[(200, image_height), (1100, image_height...

Optimizing the detected road markings

We have already identified road markings in our image from a series of points in the gradient image using the Hough transform detection algorithm. We then took these lines and placed them in a blank image, which we then merged with our color image. We then displayed the lines on the input image. Now, we will further optimize it.

It is important to first recognize that the lines currently displayed correspond to the section that exceeded the voting threshold. They were voted as the lines that best described the data. Instead of having multiple lines, as seen on the left line in our image, we will now average out their slopes and y-intercepts into a single line that traces both of the lanes.

We will do this by adding two new functions to the code make_coordinates and average_slope_intercept:

1. Import the required libraries:

In[1]: import cv2
In[2]: import numpy as np
In[3]: import matplotlib.pyplot...

Detecting road markings in a video

In this section, we will detect road markings in a video. To do so, we will use OpenCV:

1. First, import the required libraries:

In[1]: import cv2
In[2]: import numpy as np
In[3]: import matplotlib.pyplot as plt

2. Define the make_coordinates function:

In[3]: def make_coordinates(image, line_parameters):
          slope, intercept = line_parameters
          y1 = image.shape[0]
          y2 = int(y1*(3/5))
          x1 = int((y1- intercept)/slope)
          x2 = int((y2 - intercept)/slope)
          return np.array([x1, y1, x2, y2])

3. Define the average_slope_intercept function:

In[4]: def average_slope_intercept(image, lines):
          left_fit = []
          right_fit = []
          for line in lines:
            x1, y1, x2, y2 = line.reshape(4)
            parameter = np.polyfit((x1, x2), (y1, y2), 1)
            slope = parameter[0]
            intercept = parameter[1]
            if slope < 0:
              left_fit.append((slope, intercept))
  ...

Summary

This chapter explained how to find road markings in images and videos using OpenCV techniques. This is a starter project where we have implemented a road-marking detection software pipeline using OpenCV. We have applied various techniques, such as canny edge detection, the Hough transform, and grayscale conversion, which we covered in Chapter 4, Computer Vision for Self-Driving Cars. 

In the next chapter, we will look at how to use deep learning to handle image data, starting with convolutional neural networks. We will learn about the underlying concepts of convolution neural networks and implement a traffic sign classifier.