Chapter 10. Policy Gradient Methods

In the final chapter of this book, we're going to introduce algorithms that directly optimize the policy network in reinforcement learning. These algorithms are collectively referred to as policy gradient methods. Since the policy network is directly optimized during training, the policy gradient methods belong to the family of on-policy reinforcement learning algorithms. Like value-based methods that we discussed in Chapter 9, Deep Reinforcement Learning, policy gradient methods can also be implemented as deep reinforcement learning algorithms.

A fundamental motivation in studying the policy gradient methods is addressing the limitations of Q-Learning. We'll recall that Q-Learning is about selecting the action that maximizes the value of the state. With Q function, we're able to determine the policy that enables the agent to decide on which action to take for a given state. The chosen...

Policy gradient theorem

As discussed in Chapter 9, Deep Reinforcement Learning, in Reinforcement Learning the agent is situated in an environment that is in state s_t', an element of state space . The state space may be discrete or continuous. The agent takes an action a_t from the action space by obeying the policy, . may be discrete or continuous. Because of executing the action a_t, the agent receives a reward r _t+1 and the environment transitions to a new state s _t+1. The new state is dependent only on the current state and action. The goal of the agent is to learn an optimal policy that maximizes the return from all the states:

(Equation 9.1.1)

The return, , is defined as the discounted cumulative reward from time t until the end of the episode or when the terminal state is reached:

(Equation 9.1.2)

From Equation 9.1.2, the return can also be interpreted as a value of a given state by following the policy . It can be observed from Equation 9.1.1 that future...

Monte Carlo policy gradient (REINFORCE) method

The simplest policy gradient method is called REINFORCE [5], this is a Monte Carlo policy gradient method:

(Equation 10.2.1)

where R_t is the return as defined in Equation 9.1.2. R_t is an unbiased sample of  in the policy gradient theorem.

Algorithm 10.2.1 summarizes the REINFORCE algorithm [2]. REINFORCE is a Monte Carlo algorithm. It does not require knowledge of the dynamics of the environment (that is, model-free). Only experience samples, , are needed to optimally tune the parameters of the policy network, . The discount factor, , takes into consideration that rewards decrease in value as the number of steps increases. The gradient is discounted by . Gradients taken at later steps have smaller contributions. The learning rate, , is a scaling factor of the gradient update.

The parameters are updated by performing gradient ascent using the discounted gradient and learning rate. As a Monte Carlo algorithm, REINFORCE requires...

Conclusion

In this chapter, we've covered the policy gradient methods. Starting with the policy gradient theorem, we formulated four methods to train the policy network. The four methods, REINFORCE, REINFORCE with baseline, Actor-Critic, and A2C algorithms were discussed in detail. We explored how the four methods could be implemented in Keras. We then validated the algorithms by examining the number of times the agent successfully reached its goal and in terms of the total rewards received per episode.

Similar to Deep Q-Network [3] that we discussed in the previous chapter, there are several improvements that can be done on the fundamental policy gradient algorithms. For example, the most prominent one is the A3C [4] which is a multi-threaded version of A2C. This enables the agent to get exposed to different experiences simultaneously and to optimize the policy and value networks asynchronously. However, in the experiments conducted by OpenAI, https://blog.openai.com/baselines-acktr...

References

1. Sutton and Barto. Reinforcement Learning: An Introduction, http://incompleteideas.net/book/bookdraft2017nov5.pdf, (2017).
2. Mnih, Volodymyr, and others. Human-level control through deep reinforcement learning, Nature 518.7540 (2015): 529.
3. Mnih, Volodymyr, and others. Asynchronous methods for deep reinforcement learning, International conference on machine learning, 2016.
4. Williams and Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning 8.3-4 (1992): 229-256.