Distributed Distributional DDPG

The Distributed Distributional Deep Deterministic Policy Gradient (D4PG) algorithm is given as follows:

1. Initialize the critic network parameter and the actor network parameter
2. Initialize the target critic network parameter and the target actor network parameter by copying from and , respectively
3. Initialize the replay buffer
4. Launch L number of actors
5. For N number of episodes, repeat step 6
6. For each step in the episode, that is, for t = 0, . . ., T – 1:
   1. Randomly sample a minibatch of K transitions from the replay buffer
   2. Compute the target value distribution of the critic, that is,
   3. Compute the loss of the critic network and calculate the gradient as
   4. After computing gradients, update the critic network parameter using gradient descent:
   5. Compute the gradient of the actor network
   6. Update the actor network parameter by gradient ascent:
   ...