The Landscape of Reinforcement Learning

Humans and animals learn through a process of trial and error. This process is based on our reward mechanisms that provide a response to our behaviors. The goal of this process is to, through multiple repetitions, incentivize the repetition of actions which trigger positive responses, and disincentivize the repetition of actions which trigger negative ones. Through the trial and error mechanism, we learn to interact with the people and world around us, and pursue complex, meaningful goals, rather than immediate gratification.

Learning through interaction and experience is essential. Imagine having to learn to play football by only looking at other people playing it. If you took to the field to play a football match based on this learning experience, you would probably perform incredibly poorly.

This was demonstrated throughout the mid-20th...

An introduction to RL

RL is an area of machine learning that deals with sequential decision-making, aimed at reaching a desired goal. An RL problem is constituted by a decision-maker called an Agent and the physical or virtual world in which the agent interacts, is known as the Environment. The agent interacts with the environment in the form of Action which results in an effect. As a result, the environment will feedback to the agent a new State and Reward. These two signals are the consequences of the action taken by the agent. In particular, the reward is a value indicating how good or bad the action was, and the state is the current representation of the agent and the environment. This cycle is shown in the following diagram:

In this diagram the agent is represented by PacMan that based on the current state of the environment, choose which action to take. Its behavior will...

Elements of RL

As we know, an agent interacts with their environment by the means of actions. This will cause the environment to change and to feedback to the agent a reward that is proportional to the quality of the actions and the new state of the agent. Through trial and error, the agent incrementally learns the best action to take in every situation so that, in the long run, it will achieve a bigger cumulative reward. In the RL framework, the choice of the action in a particular state is done by a policy, and the cumulative reward that is achievable from that state is called the value function. In brief, if an agent wants to behave optimally, then in every situation, the policy has to select the action that will bring it to the next state with the highest value. Now, let's take a deeper look at these fundamental concepts.

...

Applications of RL

RL has been applied to a wide variety of fields, including robotics, finance, healthcare, and intelligent transportation systems. In general, they can be grouped into three major areas—automatic machines (such as autonomous vehicles, smart grids, and robotics), optimization processes (for example, planned maintenance, supply chains, and process planning) and control (for example, fault detection and quality control).

In the beginning, RL was only ever applied to simple problems, but deep RL opened the road to different problems, making it possible to deal with more complex tasks. Nowadays, deep RL has been showing some very promising results. Unfortunately, many of these breakthroughs are limited to research applications or games, and, in many situations, it is not easy to bridge the gap between purely research-oriented applications and industry problems...

Summary

RL is a goal-oriented approach to decision-making. It differs from other paradigms due to its direct interaction with the environment and for its delayed reward mechanism. The combination of RL and deep learning is very useful in problems with high-dimensional state spaces and in problems with perceptual inputs. The concepts of policy and value functions are key as they give an indication about the action to take and the quality of the states of the environment. In RL, the model of the environment is not required, but it can give additional information and, therefore, improve the quality of the policy.

Now that all the key concepts have been introduced, in the following chapters, the focus will be on actual RL algorithms. But first, in the next chapter, you will be given the grounding to develop RL algorithms using OpenAI and TensorFlow.

...

Questions

• What is RL?
• What is the end goal of an agent?
• What are the main differences between supervised learning and RL?
• What are the benefits of combining deep learning and RL?
• Where does the term "reinforcement" come from?
• What is the difference between policy and value functions?
• Can the model of an environment be learned through interacting with it?

Further reading

• For an example of a faulty reward function, refer to the following link: https://blog.openai.com/faulty-reward-functions/.
• For more information about deep RL, refer to the following link: http://karpathy.github.io/2016/05/31/rl/.