Chapter 2. Training Reinforcement Learning Agents Using OpenAI Gym

The OpenAI Gym provides a lot of virtual environments to train your reinforcement learning agents. In reinforcement learning, the most difficult task is to create the environment. This is where OpenAI Gym comes to the rescue, by providing a lot of toy game environments to provide users with a platform to train and benchmark their reinforcement learning agents.

In other words, it provides a playground for the reinforcement learning agent to learn and benchmark their performance, where the agent has to learn to navigate from the start state to the goal state without undergoing any mishaps.

Thus, in this chapter, we will be learning to understand and use environments from OpenAI Gym and trying to implement basic Q-learning and the Q-network for our agents to learn.

OpenAI Gym provides different types of environments. They are as follows:

• Classic control
• Algorithmic
• Atari
• Board games
• Box2D
• Parameter tuning
• MuJoCo
• Toy text
• Safety
• Minecraft...

The OpenAI Gym

In order to download and install OpenAI Gym, you can use any of the following options:

$ git clone https://github.com/openai/gym 
$ cd gym 
$ sudo pip install -e . # minimal install

This will do the minimum install. You can later run the following to do a full install:

$ sudo pip install -e .[all]

You can also fetch Gym as a package for different Python versions as follows:

For Python 2.7, you can use the following options:

$ sudo pip install gym              # minimal install
$ sudo pip install gym[all]         # full install
$ sudo pip install gym[atari]       #for Atari specific environment installation

For Python 3.5, you can use the following options:

$ sudo pip3 install gym              # minimal install
$ sudo pip3 install gym[all]         # full install
$ sudo pip install gym[atari]       #for Atari specific environment installation

Understanding an OpenAI Gym environment

To understand the basics of importing Gym packages, loading an environment, and other important functions...

Programming an agent using an OpenAI Gym environment

The environment considered for this section is the Frozen Lake v0. The actual documentation of the concerned environment can be found at https://gym.openai.com/envs/FrozenLake-v0/.

This environment consists of 4 x 4 grids representing a lake. Thus, we have 16 grid blocks, where each block can be a start block(S), frozen block(F), goal block(G), or a hole block(H). Thus, the objective of the agent is to learn to navigate from start to goal without falling in the hole:

import Gym
env = Gym.make('FrozenLake-v0')    #loads the environment FrozenLake-v0
env.render()                       # will output the environment and position of the agent

-------------------
SFFF
FHFH
FFFH
HFFG

At any given state, an agent has four actions to perform, which are up, down, left, and right. The reward at each step is 0 except the one leading to the goal state, then the reward would be 1. We start from the S state and our goal is to reach the G state without...

Summary

In this chapter, we learned about OpenAI Gym, including the installation of different important functions to load, render, and understand the environment state-action spaces. We learned about the Epsilon-Greedy approach as a solution to the exploration-exploitation dilemma, and tried to implement a basic Q-learning and Q-network algorithm to train a reinforcement-learning agent to navigate an environment from OpenAI Gym.

In the next chapter, we will cover the most fundamental concepts in Reinforcement Learning, which include Markov Decision Processes (MDPs), Bellman Equation, and Markov Chain Monte Carlo.