Trust Region Policy Optimization

The algorithm for Trust Region Policy Optimization (TRPO) is given as follows:

1. Initialize the policy network parameter and value network parameter
2. Generate N number of trajectories following the policy
3. Compute the return (reward-to-go) R_t
4. Compute the advantage value A_t
5. Compute the policy gradients
6. Compute using the conjugate gradient method
7. Update the policy network parameter using the update rule
8. Compute the mean squared error of the value network,
9. Update the value network parameter using gradient descent,
10. Repeat steps 2 to 9 for several iterations