Chapter 11: Machine Learning in Unity

Machine learning is the hottest buzzword in Artificial Intelligence (AI). Nowadays, everything contains (or claims to contain) some machine learning-powered AI that is supposed to improve our life: calendars, to-do apps, photo management software, every smartphone, and much more. However, even if the phrase machine learning is just a marketing gimmick most of the time, it is without question that machine learning has improved significantly in recent years. Most importantly, though, there are now plenty of tools that allow everybody to implement a learning algorithm without any previous academic-level AI knowledge.

At the moment, machine learning is not used in game development (except for applications for procedural content generation). There are many reasons for that. The main reason, though, is that a designer can't control the output of a machine learning agent, and in game design, uncontrollable outcomes often correlate to not-fun games...

Technical requirements

For this chapter, you need Unity3D 2022, Python 3.7, PyTorch, and the ML-Agents Toolkit installed on your system. Don't worry if you don't; we will go over the installation steps. You can find the example project described in this chapter in the Chapter 11 folder in the book's repository: https://github.com/PacktPublishing/Unity-Artificial-Intelligence-Programming-Fifth-Edition/tree/main/Chapter11

The Unity Machine Learning Agents Toolkit

The Unity Machine Learning Agents Toolkit (ML-Agents Toolkit) is a collection of software and plugins that help developers write autonomous game agents powered by machine learning algorithms. You can explore and download the source code at the GitHub repository at https://github.com/Unity-Technologies/ml-agents.

The ML-Agents Toolkit is based on the reinforcement learning algorithm. Simplistically, reinforcement learning is the algorithmic equivalent of training a dog. For example, if you want to teach a dog some trick, you give him a command, and then, when the dog does what you expect, you reward him. The reward tells your dog that it responded correctly to the command, and therefore, the next time it hears the same command, it will do the same thing to get a new reward.

Note

In reinforcement learning, you can also punish your agent when doing the wrong things, but in the dog-training example, I can assure you that punishment is...

Installing the ML-Agents Toolkit

As a first step, we need to download the toolkit. We can do this by cloning the repository with the following command:

git clone --branch release_19 https://github.com/Unity-Technologies/ml-agents.git

This command creates an ml-agents folder in your current folder. The ML-Agents Toolkit is composed of two main components:

• A Python package containing the Python interface for Unity and PyTorch's trainers (stored in the ml-agents folder)
• A Python package containing the interface with OpenAI Gym (https://gym.openai.com/), a toolkit for training reinforcement learning agents (stored in the gym-unity folder).

  Information

  Git is the most famous version-control application in the world. It is used to store your source code, keep track of different versions, collaborate with other people, and much more. If you are not already using Git, you should really check it out. You can download it from https://git-scm.com/.

Now, it is time...

Using the ML-Agents Toolkit – a basic example

Now that everything is installed, we can start using the ML-Agents Toolkit. First, let's explain the basic architecture of an ML-Agents scene.

An ML-Agents scene is called a learning environment. The learning environment is a standard Unity scene and contains two main elements:

• The agent: Obviously, the Agent is the central object in the ML-Agents Toolkit. An agent is an object that performs an action, receives information from the environment, and can receive rewards for actions. To create an Agent, you need to subclass the Agent class and write the behavior for the agent. For instance, if the Agent is a car, we need to write how the car is controlled by the input and how we can reward and penalize the car (for example, we can reward the vehicle for going above a certain speed and punish it when it goes off-road). A learning environment can have as many agents as you like.
• The academy: This component is a singleton...

Testing the learning environment

Before we start learning, we want to test the environment by controlling the Agents with manual input. It is very useful to debug the learning environment without wasting hours of the training process.

Fortunately, the ML-Agents Toolkit makes it very handy to control an agent with live input. We only need two steps:

1. We add the Heuristic method to the SphereAgent component. This function allows us to manually specify the values of the ActionBuffer objects. In our case, we want to add the two continuous actions to the input axes of the controller:

       public override void Heuristic(
         in ActionBuffers actionsOut) {
           var continuousActionsOut = 
             actionsOut.ContinuousActions;
           continuousActionsOut[0] = 
         ...

Training an agent

Before we can start training, we need to write a training configuration file. Open your terminal and go into any empty folder. Then, create a sphere.yaml file with the following code:

behaviors:
  MovingSphere:
    trainer_type: ppo
    hyperparameters:
      batch_size: 10
      buffer_size: 100
      learning_rate: 3.0e-4
      beta: 5.0e-4
      epsilon: 0.2
      lambd: 0.99
      num_epoch: 3
      learning_rate_schedule: linear
      beta_schedule: constant
      epsilon_schedule: linear
    network_settings:
      normalize: false
     ...

Summary

In this chapter, we barely scratched the surface of machine learning and how to use it for training Unity agents. We learned how to install Unity's official ML-Agents Toolkit, set up a learning environment, and trained the model. However, this is just a basic introduction to the ML-Agents Toolkit, and many unexplored directions are waiting for you. I encourage you to look at the ML-Agents official repository; it includes many interesting demo projects.

In the next chapter, we will wrap everything up by developing an AI agent into a more complex game demo.

Further reading

• For more information, I encourage you to check the in-depth documentation for ML-Agents in the official repository (https://github.com/Unity-Technologies/ml-agents/tree/master/docs).
• For a more in-depth (but still very accessible) introduction to reinforcement learning, there is a good article on freeCodeCamp (https://medium.freecodecamp.org/an-introduction-to-reinforcement-learning-4339519de419).
• If you are willing to go even deeper into reinforcement learning, a perfect next step is Deep Reinforcement Learning Hands-On, Second Edition, Maxim Lapan, Packt Publishing.