Chapter 8. AI Game Playing

In previous chapters, we looked at supervised learning techniques such as regression and classification, and unsupervised learning techniques such as GANs, autoencoders and generative models. In the case of supervised learning, we train the network with the expected input and output and expect it to predict the output given a new input. In the case of unsupervised learning, we show the network some input and expect it to learn the structure of the data so that it can apply this knowledge to a new input.

In this chapter, we will learn about reinforcement learning, or more specifically deep reinforcement learning, that is, the application of deep neural networks to reinforcement learning. Reinforcement learning has its roots in behavioral psychology. An agent is trained by rewarding it for correct behavior and punishing it for incorrect behavior. In the context of deep reinforcement learning, a network is shown some input and is given a positive or negative reward...

Reinforcement learning

Our objective is to build a neural network to play the game of catch. Each game starts with a ball being dropped from a random position from the top of the screen. The objective is to move a paddle at the bottom of the screen using the left and right arrow keys to catch the ball by the time it reaches the bottom. As games go, this is quite simple. At any point in time, the state of this game is given by the (x, y) coordinates of the ball and paddle. Most arcade games tend to have many more moving parts, so a general solution is to provide the entire current game screen image as the state. The following screenshot shows four consecutive screenshots of our catch game:

Astute readers might note that our problem could be modeled as a classification problem, where the input to the network are the game screen images and the output is one of three actions--move left, stay, or move right. However, this would require us to provide the network with training examples, possibly...

Example - Keras deep Q-network for catch

The objective of our game is to catch a ball released from a random location from the top of the screen with a paddle at the bottom of the screen by moving the paddle horizontally using the left and right arrow keys. The player wins if the paddle can catch the ball and loses if the balls falls off the screen before the paddle gets to it. The game has the advantage of being very simple to understand and build, and is modeled after the game of catch described by Eder Santana in his blog post (for more information refer to: Keras Plays Catch, a Single File Reinforcement Learning Example, by Eder Santana, 2017.) on deep reinforcement learning. We built the original game using Pygame (https://www.pygame.org/news), a free and open source library for building games. This game allows the player to move the paddle using the left and right arrow keys. The game is available as game.py in the code bundle for this chapter in case you want to get a feel for it...

The road ahead

In January 2016, DeepMind announced the release of AlphaGo (for more information refer to: Mastering the Game of Go with Deep Neural Networks and Tree Search, by D. Silver, Nature 529.7587, pp. 484-489, 2016), a neural network to play the game of Go. Go is regarded as a very challenging game for AIs to play, mainly because at any point in the game, there are an average of approximately 10¹⁷⁰ possible (for more information refer to: http://ai-depot.com/LogicGames/Go-Complexity.html) moves (compared with approximately 10⁵⁰ for chess). Hence determining the best move using brute force methods is computationally infeasible. At the time of publication, AlphaGo had already won 5-0 in a 5-game competition against the current European Go champion, Fan Hui. This was the first time that any computer program had defeated a human player at Go. Subsequently, in March 2016, AlphaGo won 4-1 against Lee Sedol, the world's second professional Go player.

There were several notable new ideas...

Summary

In this chapter, we have learned the concepts behind reinforcement learning, and how it can be used to build deep learning networks with Keras that learn how to play arcade games based on reward feedback. From there, we moved on to briefly discuss advances in this field, such as networks that have been taught to play harder games such as Go and Poker at a superhuman level. While game playing might seem like a frivolous application, these ideas are the first step towards general artificial intelligence, where a network learns from experience rather than large amounts of training data.