Categorical DQN

The algorithm for a categorical DQN is given as follows:

1. Initialize the main network parameter with random values
2. Initialize the target network parameter by copying the main network parameter
3. Initialize the replay buffer , the number of atoms, and also and
4. For N number of episodes, perform step 5
5. For each step in the episode, that is, for t = 0, . . ., T – 1:
   1. Feed the state s and support values to the main categorical DQN parameterized by , and get the probability value for each support value. Then compute the Q value as .
   2. After computing the Q value, select an action using the epsilon-greedy policy, that is, with probability epsilon, select a random action a and with probability 1-epsilon, select an action as .
   3. Perform the selected action and move to the next state and obtain the reward r.
   4. Store the transition information in the replay buffer .
   5. Randomly sample a transition...