Chapter 7. Robo Everything – Real Strategy Gaming

In recent times, the video gaming industry has grown at a tremendous rate. As per the 2017 year in review report by SuperData, the global gaming industry generated revenue of $108.4 billion. Newzoo, a global gaming market researcher forecast that the revenue of the video gaming industry will exceed $140 billion by 2020. 

Real-time strategy games form a sub-category of the strategy video game genre and is now gaining higher importance relative to turn-based strategy games. In this chapter, we will discuss why the AI community is behind solving real-time strategy games and how reinforcement learning is better at solve this problem statement compared to the other algorithms in terms of learning and performance.

We will cover the following topics in this chapter:

• Real-time strategy games
• Reinforcement learning and other approaches
• Reinforcement learning in RTS gaming

Real-time strategy games

The term real-time strategy (RTS) was first used by Brett Sperry as a tagline to market their game Dune II. Real-time strategy games involve the player using real-time tactics to increase assets, and save them, and utilizing them to destroy the assets of the opponent. It is associated with the many complex tactical decisions that need to be taken in a very short period of time. 

This is different from turn-based strategy games, where each opponent has time to analyze and take action while other opponents couldn't perform any actions. In real-time strategy games, the action and reaction both take place in real time, since the other entities in the environment, that is, opponents, are also active and will be performing actions simultaneously. In a real strategy game environment, there are varied forms of entities, which include players, structures, and their varied high dimensional features. Thus, the goal would be to take the optimal actions to survive in the gaming...

Reinforcement learning and other approaches

There have been many approaches devised for solving the problem of real-time strategy gaming. One of the major approaches before reinforcement learning was online case-based planning. Online case-based planning involves real-time case-based reasoning. In a case-based reasoning, a set of methods are used to learn the plans. Online case-based planning implemented this property along with the implementation of plan acquisition and execution, and that too in real time.

Online case-based planning

Case-based reasoning consists of four steps:

• Retrieve

• Reuse

• Revise

• Retain

These steps are illustrated in the following image:

Case-based reasoning

In the retrieval step, a subset of cases that are relevant to the problem are selected from the case base. In the reuse step, the solution as per the cases selected is adapted. Then, in the revision step, the adapted solution is verified through testing it in a real-world environment and observes a feedback quantifying the...

Reinforcement learning in RTS gaming

Here we will discuss how reinforcement learning algorithms can be implemented to solve the real-time strategy gaming problem. Let's recall the basic components of reinforcement learning again, they are are follows:

• States S
• Actions A
• Rewards R
• Transition model (if on-policy, not required for off-policy learning)

If these components are perceived and processed by the sensors present on the learning agent while receiving signals from the given gaming environment, then a reinforcement learning algorithm can be successfully applied. The signals perceived by the sensors can be processed to form the current environment state, predict the action as per the state information, and receive feedback, that is, reward where the action taken was good or bad. This updates that state-action pair value that is, reinforces its learning as per the feedback received.

Moreover, the higher dimension state and action spaces can be encoded to compact lower dimensions by using deep...

Summary

In this chapter, we discussed real strategy games and why researchers from the AI community are trying to solve them. We also covered the complexity and properties of real strategy games and the different traditional AI approaches, such as case-based reasoning and online case-based planning to solve them and their drawbacks. We discussed the reason behind reinforcement learning being the perfect candidate for the problem and how it is successful in fulfilling the complexity and issues related to real-time strategy games where earlier traditional AI approaches failed. We also learnt about deep autoencoders and how they can be used to reduce the dimensionality of the input data and obtain a better representation of the input.

In the next chapter, we will cover the most famous topic that brought deep reinforcement learning into the limelight and made it the flag bearer of AI algorithms, that is, Alpha Go.