At its core, quality assurance (QA) in game development isn’t much different from QA in other types of software. However, there are some QA testing aspects that are specific to games.

But, let’s first start by introducing how QA is done in modern games, when, and by whom? How is it organized in this extremely fast-paced industry? These are some of the questions we are going to answer in this chapter.

In this chapter, we will first discover the main differences between the testing of games and the testing of other types of software. Then, the reader will learn more about the importance of QA. Finally, we will go through a couple of real-world scenarios that showcase what can happen when testing goes wrong in the gaming industry.

By the end of this chapter, you will have good insights into the basics of game QA and its importance.

In this chapter, we will cover the following topics:

• Understanding the evolution of modern game testing
• Exploring the differences between software and game testing
• Why is QA important for games, especially within the agile process?
• When and how should QA testing for games be performed?

Today, QA is one of the key components of any modern software development process. It is unimaginable to release software to users without testing it first. Users now have so many choices with regard to apps, games, and digital tools, and if you release software that does not work properly or has usability issues, you risk losing many of your users. Even worse, you risk your reputation as a developer if players discover something in your game that doesn’t work and publish this information online on game forums and social media.

Therefore, QA is an important component of the development process.

QA and testing are interchangeable terms. Throughout history, humans have striven to provide quality of execution in their work – from the ancient pyramids through medieval fortresses to modern software. At its core, modern QA has its roots in medieval professional guilds, such as those for tailors, merchants, and smiths. To ensure that the quality of their products met the required quality standards, guilds implemented strict peer control that in many ways is similar to testing today. They set quality standards that guild members had to meet in order to become part of and stay in the guild. These parameters ensured that guild members everywhere provided a high level of service and in return, drove more business to them.

Modern testing is not too far from that: we test software to ensure that it meets the required quality standards and includes all implemented and approved features. Of course, these days, we have replaced the quality standards set by guilds with ones set by product owners and end users.

Modern game testing has developed along with the growth of the gaming industry. Games became widely popular in the 1980s, and they kept on evolving to various new platforms: first consoles, then home PCs, and, in the 21st century, mobile and other handheld devices. It doesn’t look like the industry is going to slow down anytime soon either. The gaming industry in 2021 was globally worth more than 180 billion USD, more than the music, TV, and film industries put together (https://www.thc-pod.com/episode/the-gaming-industry-is-now-bigger-than-movies-and-music-combined). The biggest money makers are mobile and free-to-play (F2P) games, and the top earners among them bring in over a billion USD per year (https://newzoo.com/key-numbers; https://www.statista.com/statistics/263988/top-grossing-mobile-ios-gaming-apps-ranked-by-daily-revenue/).

Games are a big business today and quality is more important than ever. With that said, there are no common standards for game QA across the industry. Every gaming studio is different and even within the same company, different teams follow different QA practices. The differences are even more significant when testing for different gaming platforms – while testing on consoles hasn’t changed too much since the beginning of game development, mobile game testing is embracing the latest trends in QA to be able to support more fast-paced development.

The terminology used in game testing is not unified (a particular term may mean completely different things in different studios), and even QA jobs might have different levels of responsibilities or completely different job descriptions from studio to studio. Taking into account all those differences, certain things remain the same. Testers generally spend months testing games repeatedly, using different approaches and shifting focus to different parts and characteristics of the game.

What would we consider to be QA today? There are many different definitions out there, but at its core, game QA is a set of testing activities, including test execution, exploration, and verification, that aims to ensure that games meet design specifications, technical quality, platform regulations, and player expectations. As we can see just from the definition, QA in gaming entails a lot of responsibilities and requires a whole range of skills. A game tester is a person who must fully understand the product’s vision, is familiar with technical risks and dependencies, can juggle conflicting priorities, knows the game better than anybody, and represents the players’ interests.

Now that you have a basic understanding of the prerequisites and best practices of QA for games, let us try and understand what makes game QA different from normal software testing. Knowledge of these differences, especially in terms of the unique aspects we test for in game development, is essential for us to master QA testing and meet the end user’s expectations within the gaming industry.

Regardless of whether our software is used in medical devices, spaceships, or the games we play, the testing methodology, if not the same, is remarkably similar. But, even if testing medical software might seem more demanding, it doesn’t mean that testing games is going to be any easier. It’s challenging to adequately perform QA on time and within budget. Usually, if development encounters problems and misses milestones, QA testing, which generally occurs toward the end of the development cycle, will also be postponed. Unfortunately, software release dates are rarely moved forward. Hence, the more delayed development is, the shorter the time span allowed for QA checks. However, the scope for testing stays the same or even grows wider. That’s why the profession of game testing is somewhat notorious for its long working hours and high stress levels.

As per their definition, games also comprise software, just like the app you use on your phone to track your steps, the software that assists pilots to fly planes more efficiently, and even the software you use to read this book! All these digital products have their differences – in terms of complexity, the programming language they use, the target audience, user experience, and so on. However, they also have lots of similarities.

Software that is used in airplanes, medical devices, and military applications is considered life-critical. That means that if such software fails for any reason, it can lead to the loss of human life. When we compare this to game software, the worst consequences of bugs in game systems are a loss of progress or, in the case of F2P games, a loss of money. While unpleasant for users, failures in game software are much less impactful.

Looking at the difference through the lens of QA, life-critical software testing is usually more rigid, takes longer, and has strict, well-defined requirements. When we talk about non-life-critical software, testing practices will very much depend on the internal company processes and software development methodology in use. Although some aspects of software testing are company- or industry-specific, the following aspects are common:

• Stability
• Scalability
• Functionality of features
• User interface (UI) look and function
• User flow and usability (UX)
• Performance under stress
• First-time user experience (FTUE)
• Localization

When testing games, in addition to the aforementioned aspects, we also test for certain specific aspects that are generally not tested in any other category of software development. These include the following:

• Fun factor
• Artificial intelligence (AI) in games
• Game physics
• Evaluation of game rules
• Level progression
• Game difficulty balancing
• Multiplayer functions
• Playthrough
• Realism
• Consistency
• Game levels
• Achievements
• Sound and music
• Voice-overs

We can see from the preceding list that although there are many crossover areas in testing, gaming software has its own specific requirements that require mastering. In this book, we will cover in detail the methodologies and practices that will help you do that.

Now that you know what makes game QA unique, let us understand why testing these unique factors is necessary to launch and maintain a successful game in the market.

By now, it’s obvious that QA is an essential part of ensuring the quality of a product. But, just how important is it? What will happen if QA is not done, or if it hasn’t been done well? There are quite a few examples of software failures that have had a big impact on end users and caused significant damage and even loss of human life.

One of the more well-known examples is the case of Stanislav Petrov, an air defense officer in the USSR, who potentially managed to avert a third world war. On September 26, 1983, while on duty, Officer Petrov received a notification from the nuclear early warning system, showing that the US had launched its nuclear missiles, attacking the USSR. Officer Petrov realized in time that the system was malfunctioning and that the alarm was false.

Gaming examples are less scary, but one that really showcases how bad game bugs can get is the case of the World of Warcraft Hakkar bug. The final boss in the Zul’Gurub raid, the Blood God Hakkar, was designed to spray Corrupted Blood when killed on enemies close to him, potentially killing weaker heroes. Unfortunately, that caused pets in the game to become poisoned and they quickly managed to spread the plague and kill thousands of players. Of course, all those deaths were digital and Blizzard, the game’s developer, reacted fast and released a patch that revived most of the slain characters. To learn more about the Hakkar bug and the impact it had on the game and the players, see https://en.wikipedia.org/wiki/Corrupted_Blood_incident.

From these couple of examples, we can already see how important role QA plays in software testing. Without skilled, efficient, and timely QA, we can lose users, revenue, and even human lives. But modern QA is important for other reasons as well. With the rise of the free-to-play (F2P) business model and the huge importance of live operations (live ops), game development had to change its methodology to accommodate fast-paced, iterative development. Most of the game development industry has now switched from the waterfall model to agile methodology.

For QA testers in gaming, that was a significant change in terms of when and how testing was executed. Instead of being done at the end of the development, QA now plays a more active part in development teams, with testers representing the players’ perspective and working closely with designers and coders as part of the team. QA testers verify issues that are highlighted by the player support team, do early tests of new features, and learn about technical risks from developers. From being purely quality gatekeepers and critics, in modern game development, QA testers are part of a team that has an accurate understanding of how users will really interact with the game.

While having QA functions embedded into the team will not replace proper playtests, it does give developers early insights into how a player will perceive certain features or changes in the game. This can make product development more efficient and help avoid making expensive design and balancing mistakes that can cause lots of harm to the end product. And while not all agile teams will have embedded QA, the ones that do usually make sure that they take full advantage of the intimate player knowledge that the QA specialists hold, as well as their deep insight into game strengths and weaknesses. This reliance on QA specialists or teams not only makes QA important to prevent games going live with bugs such as Hakkar but also helps make for more fun, engaging, and at the end of the day, more profitable games.

There are lots of discussions about when QA is executed. Traditionally, QA was always done toward the end of development. This was especially true when using a waterfall system. Following the waterfall development rules, QA testing is performed only after development has concluded and the finished software is handed over to the testing team. The following figure shows how the waterfall model follows specific steps in a particular order:

Figure 1.1 – Waterfall system

In a waterfall system, testing is done only when development is finalized. However, with the rise of agile methodology, gaming studios have gradually started to adopt this method of development as well. Furthermore, with the explosion of mobile gaming and F2P games, it became obvious that agile development was necessary for efficient live ops. This required that we change the way we test games. Handing the product to the testing team when “done” became impossible, because modern F2P games never end, and hence development is never finished.

When should QA for games be performed?

Game development has evolved into an iterative model, where we develop smaller chunks of code that are tested as they are developed. As we can see in the following figure, agile development is iterative – small, usable pieces of code are frequently released and tested:

Figure 1.2 – Iterative development

The aim of agile methodology is to develop minimum viable features that can be continuously improved in future iterations.

There are several unique aspects of testing in an agile environment:

• Testing is continuous: a QA tester still holds the position of gatekeeper and has the responsibility of validating whether a feature or the whole build is of good-enough-quality to be released to the target audience. However, besides that, QA testers also test early features, validate usability flow, develop test cases based on use case scenarios, and perform many other tasks. If your game is live, the QA team will additionally handle live testing and verification of bugs coming from player support, forums, and your game’s social media channels.
• To be able to support continuous testing, QA is part of the agile team, rather than a separate unit. It is impossible to get full insight into agile development if QA is not part of the team. In modern game development, we still often use outsourced testers for the QA-heavy stages: first-time releases, major updates, or significant changes such as the introduction of multiplayer. However, very often we have situations where we have one full-time QA team member and a scalable external QA partner.
• Testers must adapt their approach to testing based on the development stage and feature readiness. It doesn’t help to report minor bugs while testing the first version of a major feature. This can be particularly challenging for QA testers. These professionals need to have experience, skills, and great collaboration with the rest of the team to fully understand the product priorities and focus their testing effort on the right thing at the right time. In the following table, you can find a high-level overview of what the testing focus should be in each stage of game development.

  MILESTONE	TESTING FOCUS	HOW TO TEST
  PRE-PRODUCTION	User stories, scalability, game design flow	Document reviews, decision tables
  ALPHA	Core gameplay, fun factor, stability	On the selected principal device, emulators
  BETA	UI, UX, game flow, FTUE, monetization, coverage	On a wide range of target devices
  RELEASE CANDIDATE	Balancing, meta game, localization, game polish, shop and monetization, coverage	On the main supported devices

Table 1.1 – Testing focus per development phase

• Testers represent the player in the team. It’s their job not only to test the code, but also to collaborate in the development process, putting themselves in the players’ shoes. This approach requires testers to have an in-depth understanding of the target audience, its preferences, its likes and dislikes, and a solid knowledge of competitors’ games. Most of the time, the QA specialist closely collaborates and communicates with player support; testers can consequently gain precious insights into players’ gaming patterns, game preferences, and the things they find most frustrating.

Based on everything we have discussed up to here, we can freely conclude that optimally, QA testing should start as soon as we start development. Many studios still hold on to old ways of working and test only at the end of the development cycle, as it’s perceived to be cheaper. This type of testing is usually very stressful as there is rarely enough time to ensure adequate testing coverage and many serious bugs can easily creep into the live game. By having QA performed in a more iterative way, we decrease that risk. This approach is potentially much cheaper than having to fix the damage of a failed product launch.

By now, we have a better understanding of when games should be tested. But, how should the testing be undertaken?

How should QA testing for games be performed?

When we start thinking about testing, it’s important to have some kind of understanding of what the desired outcome of our test is. That will allow us to compare the actual result of the test with the desired outcome. For some tests, this might be obvious – for example, the test case open settings and mute the sound has the obvious desired outcome that the game is muted.

For other scenarios, however, it’s not so simple. If we investigate a test case such as check whether game loads, it might seem deceivingly obvious: of course, the game should load! But, how long should the loading process take? Should there be any sort of loading indicator? Will the audio be on or off while loading? As we can see, it’s not necessarily that simple to determine what the desired outcome is. This test case is something my team and I had to experience ourselves to find the answer to. It took our team almost a week, while also researching competitors, to finally agree on what the expected outcome of that particular test case should be. The desired outcome of a test case is also sometimes called the test oracle.

More often than not, we also use test cases for testing. Even if they are not a mandatory part of game testing, they are very frequently used across various gaming studios. We will study test cases in depth later in the book, but for now, it’s sufficient to become familiar with their general definition. Test cases are written instructions that list specific steps for how testing should be performed.

Besides test cases, testers can execute tests using different types of documentation. Many teams create their own product test plans, which are detailed, carefully planned documents describing all aspects of testing that will be undertaken for a specific game. Test plans usually don’t contain test cases themselves, but they specify test case tools, who created them, how, when, and based on what. The plan might also contain links to the test case repository.

If your team has fully embraced agile development practices, you might also use test charters. Test charters are less formal documents compared to test cases. They generally contain information about what the testing goal is and some ideas and approaches regarding how to reach it.

Lastly, let’s talk about who is doing the testing. By now, you have probably realized that we are using tester and QA specialist/tester as interchangeable terms. There are really no differences between the two, but, as we don’t have a unified industry terminology, there might be some differences between the two, depending on how a particular studio interprets testing tasks. However, besides professional QA personnel, there are other people who perform testing as well:

• Developers
• Members of your development team apart from developers (designers, artists, etc.)
• The marketing team
• Product managers
• Players

It’s part of the developer’s job to do testing as well, although they use a very different methodology from testers. Developer testing focuses on what is called white-box testing. This implies testing mostly focused on unit tests and code reviews. While these tests are exceptionally useful, they have slightly different purposes compared to the tests performed by QA teams.

In modern agile teams, most of the team performs at least some type of testing. Designers review the logic and validity of their designs. Artists check how their work looks and feels in the game. UX designers review any changes in the user flow. The producer will have an overall look at all the components, trying to assess the level of completion and early build quality. All of these help the QA team to get a better picture of where the specific risks are and focus their efforts in the right place at the right time.

When we talk about marketing, we primarily talk about player support. It is wise to keep the player support team informed well in advance about any game developments and provide them with design documents and access to early builds. That will help them understand the game better and address customer comments and complaints properly. As they are the people spending the most time with the players, very often, you get valuable feedback from those early play sessions.

Lastly, players themselves are also testers, whether we like it or not. They will play the game in all the possible ways they see fit and report bugs through player support, forums, or social media. You might ask, if they test it themselves, why would we bother paying specialists to do it? This is because, usually, when a player notices bugs in the game, it’s already too late, especially if the bug is a serious one. We might already have lost part of our players or a significant amount of revenue.

However, there are ways of utilizing players’ testing in a more formal way. We do that through beta testing or playtests. In these types of tests, we allow smaller, often specifically selected groups of players to play early builds of the game. The primary goal of these tests is to get an idea of how players will receive the game and their likes and dislikes, but they also very often yield useful bugs.

In this chapter, we learned about some of the basics of modern game testing. This will help you to easily transition into modern, fast-paced game teams and adapt more quickly to agile teams. You also learned about the main differences between testing software and games, which lays the foundation for what comes next: practical QA steps and the challenges that come with it.