Each application is different, and each company is structured differently, with different resources and budgets. There is no ‘one size fits all’ solution with respect to quality. Instead, we can think about test architecture as an escalator on which each time you want to focus on reaching the next step to the one you are on now. However, to understand how to get to the next level, you first need to understand the level you are currently on.

In this chapter we are going to cover the following main topics:

• Figuring out which quality assurance level your company has
• How to get to the next quality assurance level
• Identifying priorities, improvement areas, and skills needed to reach the next level

None! Take this chapter as your guiding star in testing and in this book. I hope that as you and your company progress in quality, the chapter will have a different meaning each time you read it.

Tips for absolute beginners

Whether you are a beginner at testing or even if you have an intermediate or advanced level of knowledge in testing, there are many ways in which you can find inspiration and knowledge: reading books and articles, attending talks, meetups, and conferences, attending courses, doing projects, participating in hackathons, working on projects, or playing around.

The test community is very friendly, and there are always people willing to help others.

The tools are already around those who are really committed to learning; the greatest difficulty is picking which one to use at any given time and being honest with yourself about how much time you are committed to spending on that.

This book might include concepts that are obvious to me, but you might be unaware of them. If that is the case, feel free to use the help of the references or search for those concepts before continuing to read.

I hope this book helps you find your inspiration and interesting projects to try out. Keep in mind Mark Twain’s famous words, “The secret of getting ahead is getting started.”

Getting started with a company’s testing requires making a series of decisions that could be hard to change later on. To be able to determine the best system for measuring and delivering quality for a company, you can start by asking some clarifying questions.

I also suggest you ask them whether you are about to join a new company or team to set the basis for your job and understand what to expect of the position and whether you will be able to grow professionally and help the company given its circumstances.

What development process is being followed?

Understanding the development process will help you understand how to test it better. Although companies are tending to move towards agile (for the reasons I explain next), there might also be companies that are perfectly content to be working with the waterfall system.

Waterfall

This sort of process is less common nowadays, but it is still in practice, for example, when systems are delivered in conjunction with hardware, or there are no expectations over new versions.

Testing a waterfall system usually implies managing a lengthy list of tests that are added up per feature and establishing planning for executing all those tests before launching the system to the user. We want to test everything that we can think of, and that can cover something that might already be covered with prior tests.

In these systems, it is especially expensive to find a bug in production as it might give a bad name to the product, could end up being dangerous for the end user, and might require building a new batch containing the product fix and having to re-deliver the product for free. You can see some examples of this in the Further reading section at the end of this chapter.

The tests are usually handled by one or more teams of experts, and they could be separated from the developers that built the deliverable (except for unit testing).

Although this is not so strict in a waterfall approach, starting testing early is always recommended. This is a lesson that we have learned from agile, as the earlier bugs are found, the easier they are to debug.

Agile

Although delivering bugs to the customers is something that could damage a company’s reputation and should be avoided at all costs, companies working in agile can afford to have minor bugs, provided they can be fixed quickly and seamlessly. It is important that the user has a platform to report bugs and issues if some of them reach the users.

Not delivering quickly enough could give advantages to competitors, lose clients’ interest, or even make the product irrelevant to the market. Therefore, the most important thing is to be able to balance the quality of the product with the quality of delivery time. This could mean having fewer but smarter tests. Testing something that was tested before will slow down each release. Since we have more frequent releases this could add up time. That said, every time an issue reaches production, it is important to make sure there is a new test in place that will cover this issue next time, while thinking about other tests we could have missed.

API testing gains importance over end-to-end testing in this sort of system (more on this in Chapter 4, The Secret Passages of the Test Pyramid – The Top of the Pyramid). This does not mean that long end-to-end, exploratory, or regression tests should not be done, but we can make use of the API to cover some of them (as we will see in detail in Chapter 4). Visual testing is also gaining importance in these systems.

To achieve more testing, the focus turns to deployment, as we could use servers that are out of reach for the users to do more extensive testing before deploying to them.

The relationship between developers building the feature and developers that are building automation or other members of the test team should be tight in this sort of system to avoid test repetition.

Fake agile

Many companies have moved from a waterfall approach to a fast waterfall that looks a lot like ‘agile,’ but it is not really ‘agile.’ In the process of the movement, we can see a lot of testing being carried over and trying to fit it into a smaller period for sign-off before release. These companies can be identified by having an impatient scrum expert or a project manager asking whether the test case was done and signed off every 5 minutes after the deployment had finished.

The best solution for this is to work alongside dev-ops or developers in charge of the deployments to set up a server system that would allow for more intensive, safer testing and figures out a way of cleaning the list of existing tests. In other words, we want to turn this into an agile system.

What coding language to use?

Unit tests should generally live in the same project of the class they test, using the same programming language.

If there are no tests in place yet, the advice would be to implement them in the same programming language as the developers are using. The reason is that if you turn to combined engineering or shift left in the future, the developers can take ownership of the tests. Sometimes this is difficult because you find that they are using different languages for the frontend and backend.

If there are any tests in place, the best thing is to follow along with what is there unless there is a major reason to rewrite them.

If there are too many tests in place that are not well maintained, then there are two ways of approaching this; refactoring or writing from scratch. Depending on how badly or well-kept the current tests are and how reliable their executions are, it might be easier to do the former or the latter.Here is a graph to help you decide which test code programming language to use:

Figure 1.1: Identify the most suitable test code programming language for you

The way the code is shared and controlled also tells you a lot about the system required, as does the task management, bug tracking, and logging or dashboarding systems.

Finally, if technologies used for development are changing, it would be a good moment to change the testing approach, and it is also crucial to have stable tests to make sure these changes do not affect the users.

What is the relationship between developers and testers?

As discussed in the previous sections, if developers and testers are well aligned, test repetition decreases, but independence might be affected.

Do they sit together in the office?

Being physically near shows that the company cares for communication between development and testing. If the teams sit separately, this could start to form an us and them mentality that could be seen in the product afterward.

For companies that allow remote working, the question would be whether they are in different teams or under the same manager.

Do they perform common code reviews?

Having a common code review system could result in finding bugs earlier. However, there is a general concern that having too many people signing off on a code review might slow it down or that the test team might not understand the code. I suggest that at least the unit tests are reviewed by someone with experience in testing, as code coverage tools could be easily deceived.

This also can have other benefits as all team members can improve their coding and testing skills.

What is the size of the company?

Bigger and well-established companies can afford to have specific teams or people dedicated to tool creation, automation, and quality.

On the other hand, startups struggle most to budget dedicated experts and prefer to find people who can work in multiple areas and adapt to each situation. They might prefer to deliver more bugs in exchange for quicker proof of concepts and faster development. You could argue that developing bugs and having to spend time figuring out what is wrong and where, is not at all faster than developing with superior quality from the get-go, but some of these companies have few, if any, users and are working iteratively to find the right product rather than focusing on building a product right.

The right time for a startup or product to start integrating testing earlier is when the product is about to be launched to users or when the focus changes to building the product right. Unfortunately, by this time, it might be difficult for a tester to start working on automation, as they did not participate in the feature definition and development plan, which could result in testability problems. It will also be harder to write the unit tests then. Therefore, the advice is always to incorporate testing into the development, make it a habit, and get the right set of code reviews and structural automation in place. In this way, the time added will be unnoticeable. We will talk more about unit testing, testability, and the benefits of developing with a test mentality in the next chapter.

Who writes unit tests?

This question might sound redundant, as nowadays test-driven development (TDD) is quite common, and most people understand the importance of unit testing and finding issues as early as possible. Nonetheless, this is not always granted, and some companies prefer someone other than the person that built the code to write all code for testing, including unit testing if needed. If that is the case, the test framework should include unit testing, and these tests might reside in a different place than the feature code.

What are the key test roles?

Some companies might have quality assurance (QA) people, test automation roles, and tool creators, or it might all be done by the developers, with sometimes QA being done by the product owners.

Not all companies assign the name software developer engineer in test (SDET) or QA to the same role. Therefore, it is important that you understand what each role involves and what the expected responsibilities are to be able to help achieve better quality.

It is also important to understand the number of people that are working exclusively on quality versus development, to understand the amount of automation that will be needed.

What does the app or company do?

This might seem obvious, but sometimes it is taken for granted. Depending on what the app or company does, you can think of the right and highest priority type of testing for it. For example, if the app deals with payments, localization and currency, testing is crucial.

How many users do you have or plan to have?

The current versus potential growth is as important for testing as it is for development. Understanding the amount of testing that might be done and planning the best ways of debugging issues found during testing is not always done the same way for smaller projects as for bigger ones. Similarly, we should plan the right testing environments, which might need dedicated servers or other resources. There could be specific problems, such as multi-engagement across users, and performance testing could be of higher importance.

Questions about the development

Having clarity about the way development is done in the company is highly important to create appropriate tests. We looked at development techniques before, but within these techniques, there could be differences in how the features are created. The following questions could also help clarify the quality process.

How frequent are the project iterations?

The more frequent the iterations, the less time to perform testing. If iterations are very frequent and we find the need to do exhaustive testing, we need to be creative about it. For example, having some production servers disconnected for the users with the new code and applying testing there while other servers are being built before distributing them to the users.

How long does it take to verify builds, and what is the goal?

Continuous integration/continuous delivery (CI/CD) is the most automated way of doing this, but the tests could have different degrees of extensiveness depending on the goal.

Is the priority aligned within the company? Per build or per feature?

Is there a clear list of features per build? Is there a straightforward way of tracking which old builds could be affected by new ones? If not, regression tests will need to be more extensive (it is good practice to regression test everything but there is not always enough time for it).

As you can see, successfully defining the right quality process requires a lot of different inputs. There could be more parameters that could help gain a better understanding of the particular needs of a company or project for quality, but by now, you should have a better idea of what works best for each case. In the next section, we will see three examples of common traits that I have seen in the past and analyze how they could be improved. Additionally, you can see a self-assessment in the Appendix.

In my experience, when someone asks for help getting started with testing, there are three statuses the company could be at, and therefore three things that might be needed:

1. Convincing upper management of the need to test
2. Convincing the other team members
3. Getting a full framework up and running

Convincing upper management of the need to test

As much as you are convinced that a particular set of tests are important if management believes otherwise, all testing efforts would land outside of expectations for the feature delivery, impeding the proper planning and delivery of tests. The first step before building a test framework or incorporating a new type of test is communication.

Managers’ and product owners’ main concerns usually center on time and budget. Addressing them is not always easy, and it highly depends on each case. The most important question is “why do we test?” or “why do we need this test in particular?” Make sure you can explain it and that you can add data to support the case.

Convincing other team members

Similarly to convincing upper management, other team members might understand the need to test but not agree on where and how to do so.

If there is no test team in place, the best thing would be for the developers to start getting into testing by incorporating unit testing and TDD.

It is always nice to have someone with test experience to orchestrate tests, educate the team about testing, and start creating a test system for end-to-end and other sorts of testing or even perform some degree of manual checks. The same people or different ones could do this.

If the developers have never created any tests and/or are accustomed to other people verifying everything for them, it might be difficult to educate them on the use of unit testing and testability. But, as before, having enough data could help in the path of reasoning, even if that means you write some unit tests to prove their need.

If the issue is that they do not consider testing challenging enough, try to make them passionate about it. Maybe find some challenging test problems; there are some of them in this book.

Lastly, it might be that unit testing and mocking are taking as much time or longer than the time spent on developing the feature. If this is the case, consider automating some of it. (Yes! This is possible, and we will dive deeper into this in the next chapter.)

Getting a full framework up and running

This is required when the company has some unit testing and/or manual testing in place but still does not have a test framework for a particular test type. Alternatively, they might have one but feel insecure about the current tests and want to make sure they are not missing anything to improve their system.

I will be explaining how to set some of them up throughout this book, but there is plenty of information online, comparisons, trials, and opinions, so do not feel obliged to set up the same frameworks that are here; pick the ones that suit your system the best.

You should now have a better idea about the possibilities of testing around companies and applications. In this section, we are going to review some methods for improving each of the situations.

Identifying improvement areas

The test mentality has a lot to do with asking questions and being curious. We started the chapter asking some questions, and here are some more to help you discover areas in which you could improve around quality:

Is there any repetition that could be reduced? If so, what? How could you automate those processes?

How many tests do you have of each type? We will be talking about the test pyramid in the following chapters; what does yours look like? Are there any other types of tests that could be beneficial for you? Are there any tests that are at the wrong pyramid level (or in more than one)?

How long does it take to sign off for each deployment? The ideal time should be under 15 minutes (which does not mean you cannot test further in other environments prior to or after deployment).

How much do you rely on your current tests? Are we testing what we should be testing? For example, I’ve frequently seen tests related to downloading documents from the browser. Unless your app is a browser, you should trust the download works properly. Is there another way of testing that document without going through the pain of downloading it? Are there other tests in which you are testing functionality that you do not need to test?

How much do you trust in your tests? If tests are not reliable or out of date, are they adding any value? Keeping the system clean is highly recommended. If you have a version control system, if a feature ever comes back, you should be able to retrieve the old test code from the version history.

Are you using any tools to track the issues and understand where we need or do not need tests?

Do you have the right tests for the right time? It is important to understand what tests to run and when to run each throughout development. We will discuss this further in the test pyramid chapters. We should also make sure we understand why we are testing or why we need a particular test. Avoid failing to do something just because other companies are doing it or imposing a set of tests that are not needed yet.

Lastly, if it is still hard to discern when something is needed, I highly recommend talking to a professional; some people would consider a short consultation for more tailored advice.

Building the right team – testing roles and skills

Let us just take a bit of time defining testing roles as I have found that companies do not seem to agree about their definitions, and it is important to understand what people I am referring to throughout the book. I will also add some tips to help each of the roles grow in Chapter 12, Taking Your Testing to the Next Level.

Having a test expert help figure out the maturity of the company and what is needed to improve the quality of the product is particularly important. Test managers and test architects should be in distinct positions. However, not all companies need both positions, and sometimes, the job can be done by the same person. In some cases, automation is performed by developers, other times by developers in test, and QA, they are even called “automators” (which I believe to be a made-up word).

Rather than thinking of the following as “job positions,” you could also consider them “roles” that can be performed by different professionals as needed.

Test manager

A test manager makes sure the tests are performed correctly (created by the test or dev team) and good practices are implemented. They need to understand how and what to look for in the quality area. The position requires deep knowledge of testing and people skills.

Test architect

The architect designs the frameworks and tools to be used for testing and can give practical development advice to people building test code. This position requires deep technical knowledge and experience in planning and coding tools from the start while having deep knowledge about testing. Sometimes this position is covered by an SDET.

Software development engineer

Software development engineers (SDEs) are also known as developers. They are the people in charge of building features and, depending on the company, in charge of more or less of the test code.

Manual testers

Some people refer to manual testers as QA testers. They are knowledgeable and passionate about applications, issues that could arise, and test methodologies. However, in some companies, QA testers also write some automation (generally for the user interface (UI)). Some companies invest in teaching automation, providing tooling to help them achieve more in less time, including the use of automated behavior-driven development (BDD) to turn the test definition into code, and visual testing in which UI screens are compared automatically with an expected image.

SDET

SDETs are a rare species. They have the dichotomy of being developers with a testing mentality/passion.

Being stuck in writing test code could be frustrating for most developers as it is a repetitive task, not always challenging. When a company uses SDET to define the role that I am here referring to as QA, some people find themselves in that position, expected just to write automation code, have an unpleasant experience, and move away from the title.

Instead, they should be empowered to identify processes to automate and tools they could write to keep improving their programming skills.

Many companies are starting to join a movement called “shift left” or “combined engineering” in which SDE and SDET are combined into the “software engineer” role and work on all tasks related to coding, even test coding.

DevOps

DevOps is a combination of developers and operations. A while back, tasks related to servers, deployments, and networks were done by a specialized team, sometimes referred to as “systems engineers.” Most of the time, this also included a degree of system testing and even security.

As more programming languages, tools, technologies, and techniques started to develop, roles started to get increasingly specialized. However, the issues discovered by this team were usually difficult for them to fix, as they were not part of the development and needed an understanding of the code and features that they were not familiar with.

Here was where DevOps was introduced, in other words, developers doing this job for their team rather than across the company. In some companies, the “ops” bit is taken for granted and eliminated from the word, which I will do throughout the book.

Other terms

Other terms related to testing are systems engineers or system verification testers (SVTs) (like “ops” but with more test knowledge and debugging capabilities), functional verification testers (FVTs) (a rather old term that involves end-to-end front and backend automation and testing), and integration testers.

For simplicity, they will all be referred to as SDETs in this book, even though there might be SDETs specialized in some areas as developers are.

Scaling

Horizontal scaling means that you add more machines to your testing. For example, using the cloud to test in different systems (more web browsers, mobile versus desktop, and different operating systems).

Vertical scaling means that you scale by adding more types of tests (security, accessibility, and performance). We will talk about the “test pyramid,” where each test falls on it, and what percentage of the time spent in testing should be done in each of them.

Identifying the type of testing you need and what systems should be covered should be part of an architecture design. We will talk about diverse types of testing and tools for horizontal scaling in Chapter 2, Chapter 3, and Chapter 4.

Automating the automation

Time is the highest-valued currency you have; it is something you can never get back. This is one of the reasons why I like automation, so I can save time (mine or other people’s). And by automation, I do not just mean test automation but any repetitive process. That is what I mean by “automating the automation.”

In most companies, the “task automation experts” are the SDETs. That said, if you are a developer or a QA tester, you could highly benefit from this practice.

I have identified some basic steps for automating anything:

1. Recognizing automatable tasks:

The first step to automating something is to think about repetitive tasks. Then, you should consider how long it would take to automate that task and calculate how much time it would save if it were automated.

You can tell when a company is doing well on their test automation when you see this thought process translated into it, rather than automating as many things as possible as some proof of skills or performance. The same concept can be extrapolated to any other repetitive tasks, including, of course, automation.

2. Write some code that does that task for you:

Once you have a clear picture of the steps involved in the repetitive task, you should also have an idea of how to automate those steps. The exact language, style, and patterns are up to you to define.

3. Identify when the code needs to be executed:

Sometimes we want to execute the code after something else happens. For example, testing automatically after a feature has been developed is common. Or, we can have automation that depends on some trigger, such as a page being refreshed or an application launched. Other times, we want the execution to happen at a certain point in time, for example, every morning. That could be automated with a cron job (also known as scheduled tasks) or services.

4. Identify success measures:

The next step is to identify our gain from this automation. What do we need to achieve with it? What is our best result metric? Sometimes, to verify that the automation has been executed and to check its success, we rely on logging. However, checking the logs could also be considered a manual task, and we will need to make sure we automate it if that is the case.

I suggest creating an alert if something has gone wrong, for example, a test case failing. We may also have a notification that everything has gone well, too, just to verify it has worked. An email with the details, a text message, or even a phone call could all be ways of automating the logs (more details on notifications in Chapter 11, How to Test a Time Machine (and Other Hard-to-Test Applications).

Where to start?

When there are a lot of things a company or team could improve or even automate, I have a formula for automation that we can apply to other things:

(Time spent in doing the task manually + monetary value (based on the cost of potential issues)) / (time that will take to build the automation).

I argue that we could also think of it this way:

(Time spent in doing something before a change + monetary value (how much the task is needed)) / (time that will take to implement the change).

For “time spent,” I mean total time, if there are three steps, we need to multiply it by the potential times those steps will be repeated. Of course, there might be other factors to consider as well, for example, the scalability of current versus future solutions and people impacted by this, but I found this to work for general cases and as a baseline.

Moving on

Imagine that all the feasible options in this chapter are checked and implemented, and one or more test frameworks are set up. The team is managing the new tests, which are carefully planned and executed as part of the CI/CD structure. The analytics are set up and create alerts for any problems or improvements. So, can you ever be done with testing?

I presume most people in quality will jump straight away to this answer: “you are never done with testing.” However, in the same way as you might “be done” with development, you need to “be done” with testing. It is normal if you are not 100% happy about your project, but you have done your best, and (usually due to budget) maintenance is all there is left, at least for that project (and until some cool tool or technology comes along).

What to do then? As with development, at this point, we can move on to the next project. Of course, someone should stay to fix potential issues and provide support, but you should not need such a big team or even the same skills for this. Not worse or better, simply different. Some people enjoy being at the beginning of the project, and others prefer maintaining them.

Development and testing go hand in hand; when there are fewer changes in development, there are fewer changes in test code.

In this chapter, we had an introductory overview of the test architecture and analyzed how different projects could be built and how that could affect the test architecture. Since every company is different, we need to take into account all the possible inputs to produce the right set of testing. We can conclude that as development is intertwined with testing, decisions should be made for both of them at the same time.

Then we discussed ways of achieving and improving the test architecture depending on where the project is at that point in time. We considered what can be automated and how, and what process automation follows.

We identified the different roles that can participate in quality, their responsibilities, and their skills.

In the next chapter, we will discuss what is called “the test pyramid” and we will analyze its base. The base of the test pyramid is usually handled by developers, so they may be the most interested in this chapter, although SDETs and QA testers interested in growing in their careers might find automation that they could do in this section.

In the two chapters after the next, we will talk about the middle and the top of “the test pyramid,” discussing in detail different test types and some tips and tricks around them.

Two examples of critical defects can be found with details at https://cloudwars.co/ibm/bank-of-america-cloud-slash-billions-in-it-costs/ and https://techtrends.com/2019/04/19/a-software-bug-caused-boeings-new-plane-to-crash-twice/.