Chapter 18: Challenges and Future Directions in Reinforcement Learning

In this last chapter, we summarize our journey that is coming to an end in this book: You have done a lot, so think of this as a celebration and a bird eye view of your achievement! On the other hand, when you take your learnings to use reinforcement learning in real-world problems, you will likely encounter many challenges. Thankfully, deep reinforcement learning is a fast-moving field with a lot of progress to address those challenges. We have already mentioned most of them in the book and implemented solution approaches. In this chapter, we will recap what those challenges and corresponding future directions in RL are. We will wrap up the chapter and the book by going over some resources and strategies for you to deepen your expertise in RL.

So, here is what you will read in this chapter:

• What you have achieved with this book
• Challenges and future directions
• Suggestions for aspiring reinforcement...

What you have achieved with this book

First of all, congratulations! You have come a long way to go beyond the fundamentals and to acquire the skills and the mindset to apply reinforcement learning in real-world. Here is what we have done together in this book:

• We have spent a fair amount of time on bandit problems, which have tremendous number of applications in industry and academia.
• We have gone deeper into the theory than a typical applied book to strengthen your foundation in RL.
• We have covered many of the algorithms and architectures behind the most successful applications of RL.
• We have discussed advanced training strategies to get the most out of the advanced RL algorithms.
• We have done hands-on work with realistic case studies.
• Throughout this journey, we have both implemented our versions of some of the algorithms, as well as utilized libraries, such as Ray and RLlib, which power many teams and platforms at the top tech companies for their reinforcement...

Challenges and future directions

You could be wondering why we are back to talking about RL challenges after finishing an advanced-level book on this topic. Indeed, throughout the book, we presented many approaches to mitigate them. On the other hand, we cannot claim these challenges are solved. So, it is important to call them out and discuss the future directions for each in a concise list to give you a mental map and a compass to navigate through them.

Let's start our discussion with one of the most important challenges: Sample efficiency.

Sample efficiency

As you are now well aware, it takes a lot of data to train an RL model. OpenAI Five, who became a world-class player in the strategy game Dota 2, took 128,000 CPUs and 256 CPUs to train, over many months, collecting a total of 900 years' worth of game experience per day (OpenAI, 2018). RL algorithms are benchmarked on their performances after trained over 10 billion Atari frames (Kapturowski, 2019). This is...

Suggestions for aspiring reinforcement learning experts

This book is designed for an audience who already know the fundamentals of RL. Now that you have finished this book too, you are well positioned to become an expert in this field. Having said that, RL is big area; and this book is really meant to be a compass and kickstarter for you. At this point, if you decide to go deeper in RL, I will have some suggestions.

Go deeper into the theory

In machine learning, models often fail to produce expected level of performance, at least at the beginning. One big factor that will help you go beyond what comes out of the box is to have a good foundation of the math behind the algorithms you are using. This will help you better understand the limitations and assumptions of those algorithms, identify whether they conflict with the realities of the problem at hand, and give you ideas for addressing them. To this end, here is some advice:

• It is never a bad idea to deepen your understanding...

Final words

Well, it is time to wrap up. I would like to thank you for investing your time and effort in this book. I hope it has been beneficial for you. As a last word, I would like to emphasize that getting good at something takes a long time, and there is no limit to how good you can become. Nobody is expert at everything, even in subdisciplines of ML like reinforcement learning or computer vision. Don't forget that it is a marathon you need to run. Consistency and continuity of your efforts will make the difference, no matter what your goal is. I wish you my best in your journey.

References

1. Hofmann, K. (2019). Reinforcement Learning: Past, Present, and Future Perspectives. Conference on Neural Information Processing Systems, Vancouver, Canada. URL: https://slideslive.com/38922817/reinforcement-learning-past-present-and-future-perspectives
2. OpenAI (2018). OpenAI Five. OpenAI Blog. URL: https://openai.com/blog/openai-five/
3. Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, & Remi Munos (2019). Recurrent Experience Replay in Distributed Reinforcement Learning. In International Conference on Learning Representations.
4. Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, & Marcin Michalski. (2019). SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference. arXiv.org, http://arxiv.org/abs/1910.06591.
5. Jianzhun Du, Joseph Futoma, & Finale Doshi-Velez. (2020). Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs. arXiv.org, https://arxiv.org/abs/2006.16210
6. Shen...