Practical Implementation for Resolving RL Challenges

In this chapter, we will wrap up some of the concepts behind deep reinforcement learning (deep RL) algorithms that we explained in the previous chapters to give you a broad view of their use and establish a general rule for choosing the most suitable one for a given problem. Moreover, we will propose some guidelines so that you can start the development of your own deep RL algorithm. This guideline shows the steps you need to take from the start of development so that you can easily experiment without losing too much time on debugging. In the same section, we also list the most important hyperparameters to tune and additional normalization processes to take care of.

Then, we'll address the main challenges of this field by addressing issues such as stability, efficiency, and generalization. We'll use these three main...

Best practices of deep RL

Throughout this book, we covered plenty of reinforcement learning algorithms, some of which are only upgrades (for example TD3, A2C, and so on), while others were fundamentally different from the others (such as TRPO and DPG) and propose an alternative way to reach the same objective. Moreover, we addressed non-RL optimization algorithms such as imitation learning and evolution strategies to solve sequential decision-making tasks. All of these alternatives may have created confusion and you may not know exactly which algorithm is best for a particular problem. If that is the case, don't worry, as we'll now go through some rules that you can use in order to decide which is the best algorithm to use for a given task.

Also, if you implemented some of the algorithms we went through in this book, you might find it hard to put all the pieces together...

Challenges in deep RL

The efforts that have been put into the research of reinforcement learning algorithms in recent years has been huge. Especially since the introduction of the deep neural network as a function approximation, the advancement and results have been outstanding. Yet some major issues remain unsolved. These limit the applicability of RL algorithms to more extensive and interesting tasks. We are talking about the issues of stability, reproducibility, efficiency, and generalization, although scalability and the exploration problem could be added to this list.

Stability and reproducibility

Stability and reproducibility are somehow interconnected with each other as the goal is to design an algorithm that is capable...

Advanced techniques

The challenges we listed previously have no simple solutions. However, there has been an effort in trying to overcome them and to come up with novel strategies to improve efficiency, generalization, and stability. Two of the most widespread and promising techniques that focus on efficiency and generalization are unsupervised reinforcement learning and transfer learning. In most cases, these strategies work in symbiosis with the deep reinforcement learning algorithms that we developed in the previous chapters.

Unsupervised RL

Unsupervised RL is related to the usual unsupervised learning in how both methods don't use any source of supervision. While in unsupervised learning the data isn't labeled...

RL in the real world

So far, in this chapter, we went through the best practices when developing deep RL algorithms and the challenges behind RL. We also saw how unsupervised RL and meta-learning can alleviate the problem of low efficiency and bad generalization. Now, we want to show you the problems that need to be addressed when employing an RL agent in the real world, and how the gap within a simulated environment can be bridged.

Designing an agent that is capable of performing actions in the real world is demanding. But most reinforcement learning applications need to be deployed in the world. Thus, we have to understand the main challenges that we face when dealing with the complexity of the physical world and consider some useful techniques.

Facing real-world challenges

...

Future of RL and its impact on society

The first foundations of AI were built more than 50 years ago, but only in the last few years has the innovation brought by AI spread through the world as a mainstream technology. This new wave of innovation is mainly due to the evolution of deep neural networks in supervised learning systems. However, the most recent breakthrough in artificial intelligence involves reinforcement learning, and most notably, deep reinforcement learning. Results like the ones that were obtained in the game of Go and Dota highlight the impressive quality of RL algorithms that are able to show long-term planning, ability in teamwork, and discover new game strategies that are difficult to comprehend even for humans.

The remarkable results that were obtained in the simulated environments started a new wave of applications of reinforcement learning in the physical...

Summary

Throughout this book, we learned and implemented many reinforcement learning algorithms, but all this variety can be quite confusing when it comes to choosing one. For this reason, in this final chapter, we provided a rule of thumb that can be used to pick the class of RL algorithms that best fits your problem. It mainly considers the computational time and the sample efficiency of the algorithm. Furthermore, we provided some tips and tricks so that you can train and debug deep reinforcement learning algorithms better so as to make the process easier.

We also discussed the hidden challenges of reinforcement learning: stability and reproducibility, efficiency, and generalization. These are the main issues that have to be overcome in order to employ RL agents in the physical world. In fact, we detailed unsupervised reinforcement learning and transfer learning, two strategies...

Questions

1. How would you rank DQN, A2C, and ES based on their sample efficiency?
2. What would their rank be if they were rated on the training time and 100 CPUs were available?
3. Would you start debugging an RL algorithm on CartPole or MontezumaRevenge?
4. Why is it better to use multiple seeds when comparing multiple deep RL algorithms?

5. Does the intrinsic reward help with the exploration of an environment?
6. What's transfer learning?

Further reading

• For an approach that uses a pure curiosity-driven approach in the Atari games, read the paper Large-scale study of curiosity-driven learning (https://arxiv.org/pdf/1808.04355.pdf).
• For practical use of domain randomization for learning dexterous in-hand manipulation, read the paper Learning Dexterous In-Hand Manipulation (https://arxiv.org/pdf/1808.00177.pdf).
• For some work that shows how human feedback can be applied as an alternative to the reward function, read the paper Deep Reinforcement Learning from Policy-Dependent Human Feedback (https://arxiv.org/pdf/1902.04257.pdf).