Road Sign Detection Using Deep Learning

Traffic road signs are an important part of our road infrastructure. Without road signs, the percentage of road accidents would increase; drivers wouldn't know, for example, how fast they should be going or whether there are roadworks, sharp turns, or school crossings ahead. In this chapter, we are going to solve an important problem in self-driving car (SDC) technology – traffic sign detection. We will implement a model that will achieve 95% accuracy.

We are going to use the German traffic signs dataset, which is free to use. I have taken inspiration from the following publication: J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition (https://ieeexplore.ieee.org/document/6033395). In Proceedings of the IEEE International...

Dataset overview

Traffic sign detection using the German traffic sign dataset is a multi-class classification problem that contains more than 40 classes and more than 50,000 images. The physical traffic sign instances are unique within the dataset, and each real-world traffic sign only occurs once.

You can follow this link to find out more about the German traffic sign detection dataset: J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. In Proceedings of the IEEE International Joint Conference on Neural Networks, pages 1453–1460. 2011 (http://benchmark.ini.rub.de/).

Dataset structure

The training set archive is structured as follows:

• One directory per class.
• Each directory contains one Comma Separated Value (CSV) file with annotations (GT-<ClassID>.csv), as well as the training images.
• Training images are grouped by tracks.
• Each track contains 30 images of one single physical traffic sign.

Image format

The image format is structured as follows:

• The images contain one traffic sign each.
• Images contain a border of 10 % around the actual traffic sign (at least 5 pixels) to allow for edge-based approaches.
• Images are stored in PPM format – Portable, Pixmap, and P6 (http://en.wikipedia.org/wiki/Netpbm_format).
• Image sizes vary between 15 x 15 and 250 x 250 pixels.
• Images are not necessarily squared.
• The actual traffic sign is not necessarily centered within the image. This is true for images that were close to the image border in the full camera image.
• The bounding box of the traffic sign is a part of the annotations, which we will see in the following section.

The following are examples of a few classes:

• ( 0, b'Speed limit (20 km/h)') ( 1, b'Speed limit (30 km/h)')
• ( 2, b'Speed limit (50 km/h)') ( 3, b'Speed limit (60 km/h)')
• ( 4, b'Speed limit (70 km/h)') ( 5, b'Speed limit (80 km/h)')
• ( 6, b'End of...

Loading the data

In this section, we will start building and training our convolutional neural network:

1. We will start by importing the necessary libraries, including pandas, numpy, and matplotlib:

import warnings
warnings.filterwarnings("ignore")
import libraries 
import pickle
#Import Pandas for data manipulation using dataframes
import pandas as pd 
#Importing Numpy for data statistical analysis 
import numpy as np  
#Importing matplotlib for data visualisation
import matplotlib.pyplot as plt
import random

2. Next, we import three pickle files – the test, training, and validation datasets:

with open("./traffic-signs-data/train.p", mode='rb') as training_data:
    train = pickle.load(training_data)
with open("./traffic-signs-data/valid.p", mode='rb') as validation_data:
    valid = pickle.load(validation_data)
with open("./traffic-signs-data/test.p", mode='rb') as testing_data:
    test = pickle.load(testing_data...

Image exploration

In this section, we will explore the images class and see what the German traffic sign dataset looks like.

Let's look at the image and check whether it has been imported correctly:

i = 1001
plt.imshow(X_train[i]) # Show images are not shuffled
y_train[i]

The output looks like a signboard, so we can see that the image has been imported correctly:

Fig 7.1: Output of the imported image

In the following image, you can see all the classes of the German traffic sign detection dataset, which we discussed in the Image format section:

Fig 7.2: 43 image classes

In the next section, we will start the first step of the modeling process – data preparation.

Data preparation

Data preparation is an important part of any data science project. In this section, we will prepare the data for analysis. Data preparation helps us achieve better accuracy:

1. We will start by shuffling the dataset:

from sklearn.utils import shuffle
X_train, y_train = shuffle(X_train, y_train)

2. Now, we will transform the data into grayscale and normalize it:

X_train_gray = np.sum(X_train/3, axis=3, keepdims=True)
X_test_gray = np.sum(X_test/3, axis=3, keepdims=True)
X_validation_gray = np.sum(X_validation/3, axis=3, keepdims=True) 

X_train_gray_norm = (X_train_gray - 128)/128 
X_test_gray_norm = (X_test_gray - 128)/128
X_validation_gray_norm = (X_validation_gray - 128)/128

3. Next, we will check the images following the grayscale conversion:

i = 610
plt.imshow(X_train_gray[i].squeeze(), cmap='gray')
plt.figure()
plt.imshow(X_train[i])

The output image should look as follows:

Fig 7.3: Grayscale image

In the next section, we will start the model training process...

Model training

In this section, we will train our model with the same convolution neural network architecture that we used in Chapter 6, Improving the Image Classifier with CNN. Let's get started:

1. We will start by importing the keras and sklearn libraries. We are importing Sequential, Conv2D, MaxPooling2D, Dense, Flatten, Dropouts, Adam, TensorBoard, and check_ouput from Keras. We have imported train_test_split from sklearn:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from subprocess import check_output
from sklearn.model_selection import train_test_split

2. Next, we will build the following model:

cnn_model = tf.keras.Sequential()
cnn_model.add(tf.keras.layers.Conv2D(32,3, 3, input_shape = image_shape, activation='relu'))
cnn_model.add(tf.keras.layers.Conv2D(64, (3,3), activation='relu'))
cnn_model.add(tf.keras.layers.MaxPooling2D(pool_size = (2, 2)))
cnn_model.add(tf.keras.layers.Dropout(0.25))
cnn_model.add(tf.keras.layers...

Model accuracy

Once training is complete, the next step is to validate the model accuracy. Let's get started:

1.  Validate the accuracy of the model:

score = cnn_model.evaluate(X_test_gray_norm, y_test,verbose=0)
print('Test Accuracy : {:.4f}'.format(score[1]))

2. The accuracy of the model is as follows:

Test Accuracy : 0.9523

3. The training loss versus validation loss graph looks like this:

import matplotlib.pyplot as plt

history_dict = history.history
loss_values = history_dict['loss']
val_loss_values = history_dict['val_loss']
epochs = range(1, len(loss_values)+ 1)
line1 = plt.plot(epochs, val_loss_values, label = 'Validation/Test Loss')
line2 = plt.plot(epochs, loss_values, label= 'Training Loss')
plt.setp(line1, linewidth=2.0, marker = '+', markersize=10.0)
plt.setp(line2, linewidth=2.0, marker= '4', markersize=10.0)
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.grid(True)
plt.legend()
plt.show()

The...

Summary

As we already know, it is really important for an autonomous driving car to follow traffic signals. In this chapter, we have implemented traffic sign detection with 95% accuracy. This project is one of the important steps toward creating autonomous driving cars. We created a model that classifies traffic signs and identifies the most appropriate features by itself.

In the next chapter, we will learn about semantic segmentation, which is the next step toward building SDCs.