Chapter 9. Reinforcement Learning in Autonomous Driving
In this chapter, we will cover different approaches researchers are working on to make end-to-end autonomous driving possible. We have seen many companies, such as Google, Tesla, Uber, Mercedes Benz, Audi, Volvo, Bosch, and many more enter the domain of self-driving cars. For the AI community, end-to-end autonomous driving will be the next milestone to achieve on the route to artificial general intelligence (AGI).
Looking at the current trend in automotive industry, we witness the following:
- Environment and climate friendly electric cars are increasing
- Monetization through cab aggregator service and carpooling, that is, ride sharing
- Disruptive research on autonomous vehicles using AI and cloud power
Key Lego blocks of autonomous driving are as follows:
- Sensor fusion (sensors can be camera, LIDAR, RADAR, GPS, and so on)
- Object detection and classification
- Vehicular path planning—which action to take such as steer left or right, accelerate, or...
Machine learning for autonomous driving
Firstly, in order to develop an end-to-end self-driving car we must know the development process at a high-level before delving into the use of reinforcement learning in the whole process. The following is a diagram that depicts the development process:
As shown in the preceding figure, the first step of the process is the collection of sensor data. Sensors comprise a camera, LIDAR, IMU, RADAR, GPS, CAN, and many more devices that can capture the state of the vehicle as well as the surrounding environment in the best possible way. After receiving these sensory signals, they are preprocessed, aggregated, and then prepared for sending to the next process, which includes machine learning (ML) and analysis in the data center. This step of implementing ML on the prepared sensory signals is a key part, which involves state estimation from the input data, thereby modeling it, predicting the possible future actions, and finally, the planning as per the predicted...
Reinforcement learning for autonomous driving
The challenge posed by autonomous driving cannot be solved by a full supervised learning approach owing to strong interactions with the environment and multiple obstacles and maneuvers (discussed previously) in the environment. The reward mechanism of reinforcement learning has to be highly effective so that the agent is very cautious about the safety of the individual inside and all the obstacles outside, whether it's humans, animals, or any ongoing construction.
One of the approaches to rewards could be:
- Agent vehicle collides with the vehicle in front: High negative reward
- Agent vehicle maintains safer distance from both front and rear end: Positive reward
- Agent vehicle maintains unsafe distance: Moderate negative reward
- Agent vehicle is closing the distance: Negative reward
- Agent vehicle speeds up: Decreasing the positive reward as the speed increases and negative when it crosses the speed limit
Incorporating recurrent neural networks (RNNs) to...
Proposed frameworks for autonomous driving
In this section, we will discuss a proposed deep reinforcement learning framework for autonomous driving given by El Sallab et. al 2017 (https://arxiv.org/pdf/1704.02532.pdf).
The following is an architecture of end-to-end deep neural networks:
End to End training of Deep Neural Networks for Autonomous Driving by El Sallab et. al 2017 (https://arxiv.org/pdf/1704.02532.pdf)
Let's discuss the preceding architecture in detail. Inputs in this case are the aggregation of states of the environment over multiple timesteps.
The first unit of the architecture is the spatial aggregation network. It consists of two networks, each for the the following sub-processes:
- Sensor fusion
- Spatial features
The overall state includes the state of the vehicle as well as the state of the surrounding environment. The state of the vehicle includes position, geometric orientation, velocity, acceleration, current fuel left, current steering direction, and many...
DeepTraffic – MIT simulator for autonomous driving
DeepTraffic (https://selfdrivingcars.mit.edu/deeptraffic/) was created for the course MIT 6.S094: Deep Learning for Self-Driving Cars at MIT taught by Lex Fridman. Course content and assignment is public. DeepTraffic gained a lot of popularity owing to its leaderboard. With over 13,000 submissions to date, DeepTraffic is highly competitive. The users have to write their neural networks in convnet.js (a framework created by Andrej Karpathy) in the coding ground present in the link mentioned at the start of the section. The agent with the maximum average speed tops the leaderboard.
Simulations such as DeepTraffic help train different approaches to make the car agent adapt to the simulated environment quickly. Moreover, the competitive element of it adds to better submissions over time, beating the past top scores. The competition makes it fun but in the real world a student can't test their deep reinforcement learning scripts. Therefore, DeepTraffic...
In this chapter, we touched on the main concepts and challenges related to one of the biggest AI problems, that is, autonomous driving. We learned about the challenges posed by the problem and also learned the current approaches being used to make autonomous driving successful. Moreover, we went through an overview of different sub-tasks of the process, starting from receiving sensory inputs to planning. We also looked at a bit about the famous DeepTraffic simulation where you can test your neural networks to learn efficient movement patterns in heavy traffic. Autonomous driving is itself a vast evolving research topic and covering all of them is beyond the scope of this book.
In the next chapter, we will study another evolving research hotspot, using AI in finance, where we will learn how reinforcement can help in financial portfolio management.
