PPO-Clipped

The algorithm for the PPO-clipped method is given as follows:

1. Initialize the policy network parameter and value network parameter
2. Collect some N number of trajectories following the policy
3. Compute the return (reward-to-go) R_t
4. Compute the gradient of the objective function
5. Update the policy network parameter using gradient ascent,
6. Compute the mean squared error of the value network,
7. Compute the gradient of the value network
8. Update the value network parameter using gradient descent,
9. Repeat steps 2 to 8 for several iterations