Double DQN

The algorithm for double DQN is given as follows:

1. Initialize the main network parameter with random values
2. Initialize the target network parameter by copying the main network parameter
3. Initialize the replay buffer
4. For N number of episodes, repeat step 5
5. For each step in the episode, that is, for t = 0, . . ., T – 1:
   1. Observe the state s and select an action using the epsilon-greedy policy, that is, with probability epsilon, select random action a with probability 1-epsilon, and select the action as
   2. Perform the selected action and move to the next state and obtain the reward r
   3. Store the transition information in the replay buffer
   4. Randomly sample a minibatch of K transitions from the replay buffer
   5. Compute the target value, that is,
   6. Compute the loss,
   7. Compute the gradients of the loss and update the main network parameter using gradient descent:
   ...