On-Policy TD Control – SARSA

The algorithm for on-policy TD control – SARSA is given as follows:

1. Initialize the Q function Q(s, a) with random values
2. For each episode:
   1. Initialize the state s
   2. Extract a policy from Q(s, a) and select an action a to perform in the state s
   3. For each step in the episode:
      1. Perform the action a, move to the new state , and observe the reward r
      2. In the state , select the action using the epsilon-greedy policy
      3. Update the Q value to
      4. Update and (update the next state -action pair to the current state s-action a pair)
      5. If s is not the terminal state, repeat steps 1 to 5