The Markov Decision Process (MDP) provides a mathematical framework for solving the reinforcement learning (RL) problem. Almost all RL problems can be modeled as MDP. MDP is widely used for solving various optimization problems. In this chapter, we will understand what MDP is and how can we use it to solve RL problems. We will also learn about dynamic programming, which is a technique for solving complex problems in an efficient way.
In this chapter, you will learn about the following topics:
- The Markov chain and Markov process
- The Markov Decision Process
- Rewards and returns
- The Bellman equation
- Solving a Bellman equation using dynamic programming
- Solving a frozen lake problem using value and policy iteration
the agent can actually be in.
that can be performed by an agent, for moving from one state to another.
), which is the probability of moving from one state 
to another 
state by performing some action 
.
), which is the probability of a reward acquired by the agent for moving from one state 
to another state 
by performing some action 
.
), which controls the importance of immediate and future rewards. We will discuss this in detail in the upcoming sections.
is the one which yields maximum value compared to all the other value functions:
is the one that has a higher value compared to all other value functions (that is, maximum return), it will be the maximum of the Q function. So, the optimal value function can easily be computed by taking the maximum of the Q function as follows:
-- (3)
