How to Test a Time Machine (and Other Hard-to-Test Applications)

Throughout all the previous chapters, we covered different technologies, the knowledge of which can help while testing difficult apps. These concepts should cover most of our quality needs. However, sometimes it happens that we face applications and systems out of the norm, and guides as such the ones provided in the previous chapters would be insufficient for our needs.

In this chapter, we will focus on challenges that are particularly difficult to solve and provide solutions that you might find useful for your own challenges and systems.

Before we dive deep into the different solutions to ensure quality on difficult applications, let us highlight the importance of understanding what’s worth spending time and resources to automate and what’s not. If the effort spent on automation or even the solution of a problem surpasses, in the long run, the effort of a different workaround, such as performing...

Technical requirements

The first section of this chapter is merely hypothetical, included so that you understand the different approaches to testing before starting with any programming or deciding on the architecture of a system, so there are no technical requirements for it. For the rest of the chapter, some degree of programming skills is recommended to get the best out of the examples.

Whilst this chapter is written with a QA/SDET role in mind, as applications shift left, developers may also find this chapter interesting. Furthermore, if you are trying to get developers more involved in testing, this is one of the chapters you would ideally show to them, as it, hopefully, will trigger their building instincts and curiosity so that they also contribute to the creation of amazing applications to improve the quality of systems.

This chapter uses examples written in various programming languages. We recommend reviewing and working with different languages as a self-growth exercise...

How to test a time machine

You are probably wondering how we came up with the unique title of this book as well as this chapter. It all began with an idea I wrote on my blog – if a time machine were to exist, how would you test it?

This comes from a typical test interview question in which the interviewer would ask the interviewee how to test some object, and try to identify/organize the cases in different types of tests. The question might appear silly, but it actually tells a lot about the interviewee’s testing capabilities, out-of-the-box thinking, curiosity, inventive and testing knowledge.

Therefore, feel free to take a pause from reading, take a notepad, and do this exercise – if time machines were to exist, how would you design the testing for them? Start writing everything that comes to mind. Then, keep reading to see whether there was something that we or you have missed.

At the end of the day, when it comes to testing an application, you may...

Challenges in game testing

If you like playing games, testing them might sound like the most ideal, coolest job there could be. However, in reality, if your job is to manually test some screens, levels, or actions over and over again, it may be less fun than you think, and it could even spoil the game for you.

Nonetheless, working in game testing automation is possibly one of the most challenging and interesting areas related to testing (if you don’t mind a small game spoiler here and there).

Some of the reasons games are such challenging applications to work with are the uncertainty related to them and the use of specific technologies, such as high-end graphics, cross realities, and artificial intelligence, in their creation.

There are some studies out there that are worth reading in relation to uncertainty. Basically, uncertainty gives a game the fun factor. The game Tic-Tac-Toe was fun until people figured out that it is possible to find the optimal strategy to win...

Automating the impossible

There are some times that we are told that automating this and that is impossible. I personally relish this challenge. Consider that we have all these inventions and applications nowadays that years ago would have seemed impossible. We have gotten so used to some of them, such as mobile phones that allow us to connect with anybody anywhere in the world, even with real-time video, which sounded impossible many years ago.

Using AI, some apps seem to be capable of foretelling behaviors, virtual reality combines with our reality to create amazing and entertaining experiences, and the internet of things allows wristbands and watches to know whether you slept well enough, how many calories you have burned today, or even if you have washed your hands thoroughly. This all seemed like science-fiction material just a few years ago.

We are starting to see AI helping both development and testing.

Imagine that you have found something challenging and interesting...

Automating for a higher purpose

Time is one of the most valuable currencies we possess. Yet we keep wasting it and our energies doing tiresome and repetitive tasks. This is why I love automation and creating tools, not only for testing but also to help speed up the entire development process and, really, for everything in life.

Next, we will discuss a number of things you could automate, by coding or with tools, and some hints on how to do so. Some of the things in the list might be obvious to you, but many people tend to forget about them, so use this as a refresher if you already knew about them.

While it takes no computing effort to deal with it, one thing that people tend to forget is that they can automate email sorting. Creating rules to automatically sort out your email in folders might seem tiresome, but it will save you time in the long run. The steps on how to do that are different depending on the email provider that you use, so we will not review this here. However...

Last-minute tips and best practices for your automation

I am sure I will forget to mention many important things in this book (besides the ones I might not be aware of myself). Before moving on to how to architect a system and put it all together, I think it’s important to mention some best practices.

A common question asked in the testing world is how to test the uploading and downloading of files. When your application needs to download or upload a file, skip that bit during testing, and test all the way to before and after the action. You can use an API call to retrieve the file and then check for specific values within the file. You can check that a form shows an error message if a file is not attached. However, do not waste time making sure that the file is downloaded, as that is up to the browser to handle. If the browser does not download the file, the problem lies beyond your app.

We could say a similar thing about third-party applications; it is good to test the...

Architecting a system

We have seen some of the options we have whe building automation for our system, what is left is how to put our system together, as this book is about test architecture, not only its components.

There are many ways we could automate our system and many choices we could make. A good system architecture will allow you to replace components with others when needed, in such a way that the system will still work.

For example, in this chapter, we have seen how to call different programs within another program in the testReporting.py code. However, in Chapter 6, Continuous Testing – CI/CD and Other DevOps Concepts You Should Know, we did something similar as a part of a CI/CD system in the logAnalizer.py code. How shall we decide between the two ways?

For simple applications or a quick turnaround, we could go for the solution in this chapter. However, if some of the programs that are called with subprocesses were to break (or someone deliberately breaks...

Summary

In this chapter, we have seen some tips and tricks to test challenging applications, covering games, including difficult ones, non-testing automation, and even what to do with impossible-to-test applications.

We started the chapter by reviewing how to test a time machine (which gives this book its title). We saw how to approach the testing of a system in an ordered way and how to think about all levels of testing and all the testing types that we could implement in our system. This could be extrapolated to other apps, the time machine being an example.

Then, we reviewed a list of different issues that hard-to-test applications can have and some potential solutions to those issues. We continued the chapter by recalling things that we could automate that we tend to forget about, including record-and-playback tools and their advantages and disadvantages, followed by some general automation tips.

We finished the chapter by talking about test architecture (which also gives...

Further reading

• [1] The Apollo 11 code was uploaded here: https://github.com/chrislgarry/Apollo-11.
• [2] My blog is https://thetestlynx.wordpress.com; you can check the past posts section to find the one about the test machine.
• [3] For more information about the pyautogui library and how to control the mouse and keyboard with Python, see https://pyautogui.readthedocs.io/en/latest/; for the win32gui library, visit http://timgolden.me.uk/pywin32-docs/win32gui.html.
• [4] More information about the UIA tree is available here: https://learn.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-tree-overview.
• [5] Working with JSON objects can make things easier; you can find out more about the JSON library for Python here: https://docs.python.org/3/library/json.html.
• [6] WinAuto is a bit old now: https://docs.microsoft.com/en-us/windows/win32/winauto/inspect-objects. Accessibility insights is the recommended tool to inspect Windows elements at the time of...