Chapter 11. Reinforcement Learning in Robotics

So far, we have seen the advancements of reinforcement learning in AlphaGo, autonomous driving, portfolio management, and a lot more. Studies and research say that reinforcement learning can provide features of cognition such as animal behavior.

A close comparison with cognitive science would be the many successful implementations of reinforcement learning in dynamic robotic systems and autonomous driving. They have proved the theory behind applying reinforcement learning algorithms for real-time control of physical systems.

The use of neural networks in deep Q-networks and policy gradients removes the use of hand engineered policy and state representations. The direct implementation of CNNs in deep reinforcement learning and using image pixels as states instead of hand engineered features, became a widely accepted practice. The concept of mini batch training and separate primary and target networks brought success to deep reinforcement learning...

Reinforcement learning in robotics

Robotics is associated with a high level of complexity in terms of behavior, which is difficult to hand engineer nor exhaustive enough to approach a task using supervised learning. Thus, reinforcement learning provides the kind of framework to capture such complex behavior.

Any task related to robotics is represented by high dimensional, continuous state, and action spaces. The environmental state is not fully observable. Learning in simulation alone is not enough to say the reinforcement learning agent is ready for the real world. In the case of robotics, a reinforcement learning agent should experience uncertainty in the real-world scenario but it's difficult and expensive to obtain and reproduce.

Robustness is the highest priority for robotics. In normal analytics or traditional machine learning problems, minor errors in data, pre-processing, or algorithms result in a significant change in behavior, especially for dynamic tasks. Thus, robust algorithms...

Challenges in robot reinforcement learning

Applications of reinforcement learning in robotics include:

• Locomotion
• Manipulation
• Autonomous machine control

As discussed previously, in order for a reinforcement learning agent to perform better in a real-world task it should have a well-defined, domain-specific reward function, which is hard to implement. This problem is being tackled by using techniques such as apprenticeship learning. Another approach to solve the uncertainty in reward is to continuously update the reward functions as per the state so that the most optimized policy is generated. This approach is called inverse reinforcement learning.

Robot reinforcement learning is a hard problem to solve owing to many challenges. The first being continuous state-action spaces. The decision is, as per the problem statement, whether to go for DAS algorithms or CAS algorithms. This means at what granular level the robot control should be. One big challenge is the complexity of the real-world systems...

Open questions and practical challenges

As per the different challenges in reinforcement learning algorithms, they cannot be directly implemented to robotics compared to supervised learning where large scale significant progress has already been done in terms of research and better deployment.

Reinforcement learning can be introduced for various physical systems and control tasks in robotics where risk isn't very high. The reason behind this is the question of stability of a reinforcement learning model in the real-world system. All learning processes require implemented domain knowledge for better state representations and devising accurate reward functions. This requires further research and development.

Let's discuss some of the open questions for reinforcement learning algorithms that require more attention in ongoing and future research in the space of robot reinforcement learning.

Open questions

Following is a list of open, non-exhaustive questions that demand special care to deliver better...

Key takeaways

In this chapter, we have gone through the major challenges faced by reinforcement learning algorithms in the field of robotics. Therefore, the key takeaways for students who want to enter this great research domain of robot reinforcement learning are shown in the following diagram:

Summary

In this chapter, we covered the current state of reinforcement learning algorithms and the challenges in the field of robotics. We also tried to take a look at each of the challenges in detail. We also learned about the practical challenges and its proposed solutions. Cracking the solution for end-to-end robotics will be the biggest milestone for the AI community. At present, there are challenges with continuous improvements in algorithms and data processing units; however, the day we see robots doing general human tasks is not far off. In case, you want to follow-up some of the researches done in robot reinforcement learning then you would like to start with the options below:

• "Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates" by Shixiang Gu et al. 2016 (https://arxiv.org/pdf/1610.00633.pdf)
• "Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search" by Yahya et al. 2016 (https://arxiv.org/pdf/1610.00673.pdf...