The test automation tool creates a set of scripts that emulates the user action to be performed on the Application Under Test (AUT) and replays them over and over with minimal or no human intervention. Test automation is a complete process, which goes through distinct phases to automate test cases for applications under the test. The AUT can belong to any technology or domain.
Test automation should start very early in the life cycle of the project and its success depends upon the following:
Knowledge of the automation life cycle
Processes in test automation
Selection of the right tool
Test automation planning
Appropriate technical know-how about automation and test automation tools
The key driving and guiding forces for successful test automation are the automation goals, best practices, selection criteria, and attributes for tool selection and processes.
Good knowledge in these areas helps implement the test automation in an effective way.
Test automation life cycle is the process in which test automation progresses through a series of different stages. The focus of this chapter is to learn the key activities and tasks that are performed during each phase of the life cycle.
A feasibility study is an evaluation and analysis of the potential tool, which can provide the desired solution for the test automation needs of the project(s). The aim is to uncover the strengths and weaknesses of the tool(s) objectively and rationally. Generally, it precedes test automation implementation.
The feasibility study between the manual and automation effort required should be applied based on the following:
ROI (Return On Investment)
Future enhancements
Consider the test cases, automation tools, automation environment, development and maintenance of automated tests, skill sets, organizational capabilities, design, and approach of the automation for test automation planning.
The simplest way to define the ROI is ROI = Benefits/Cost.
The go/no go decision can be taken after the calculation of the ROI. The future changes in the application/service might possess many threats as well as opportunities for test automation. Leads and managers need to have a clear vision to evaluate the ROI for future enhancements.
The test manager or lead should focus on maximizing the ROI for test automation. He/she should focus on reducing the maintenance cost and achieving faster scalability of the test suite. Computation of the ROI is an important activity, which allows us to foresee many tradeoffs and governs the implementation of test automation. Since computing the ROI is out of scope for this book, this is how far we can go.
This phase is designed for successful induction of tool(s) and building the basic infrastructure needed or identifying the needs for test induction.
The key steps in test induction are listed as follows:
Identifying the technical/non-technical requirements:
The technical requirements are, for example, support for technology, tools, and scripting languages
Non-technical requirements are, for example, the goals for test automation
Evaluating the tools based on selected attributes along with their respective weightage. Tool evaluation is one of the most technically-challenging tasks when performed with limited knowledge about the tool(s).
Attributes such as user friendliness, license cost, and training cost define tool features
Assign higher weightage (a high number) to the most relevant attributes during tool selection
Compute the overall score by calculating the sum of the product of weight and value, that is, Total score = ∑ Attribute weight * Attribute value. The range of the value may vary, for example from zero to five.
Pilot run is when we create the sample scripts to verify that the selected tool is working as expected on the selected test cases. If the pilot run is successful, you can purchase the license or acquire the tool.
Test automation strategy is a plan of action designed to achieve the overall aim for test automation. It provides broad guidelines so that the stakeholder knows the approach for design and implementation of the test automation. Few organizations combine the test strategy and test plan in a single document.
Test strategy defines the following:
Planning is the process of thinking and looking ahead for activities and tasks required for test automation. The test plan defines the following:
Testing architecture
Tool installation plan
Test schedule
Roles and responsibilities
Resource requirements (hardware, software, and skilled resources)
Training requirements
Reviewing the plans for scripts and design documents
Limitations and assumptions
The test plan document tells us what is to be done for test automation, and when. This document is used by all the stakeholders and it serves as the contractual agreement among the key stakeholders.
This is the key phase where actual things get implemented. This phase requires technical skills for designing the framework and coordination skills to interact with the SMEs (Subject Matter Experts), BAs(Business Analysts), manual testers, and developers to accomplish the various tasks for test automation. The leads have to ensure that the reviews have been done and coding standards followed; the test repository structure is then created and all the resources are kept at appropriate locations. We will create the documents for understanding, using, and enhancing the test automation later.
The key steps in the design and development phase are as follows:
Analysis and script planning
Script creation (scripts can be created by recording or manually)
Creation of common and framework-specific libraries
Script enhancement (adding exception handlers, checkpoints, and so on)
Association or integration of libraries and functions with scripts
Reviews (ensure consistency, coding standards, and version controls are followed as already defined in the plan or strategy document)
Dry run and analyzing the results
Debugging the scripts and fixing the defects
In the previous phase we have performed the dry run; now it is time to deploy the scripts to the appropriate location with a well-defined structure and data. The actual execution may not be carried out by the people who developed the test scripts, as it may be executed by business users, SMEs, or other automation engineers; hence, before releasing the test scripts it should be ensured that all the required documents and automation resources are in place and reviewed by the lead or team members.
The key steps in the deployment, release, and maintenance phase are as follows:
Deployment: After the test automation scripts are deployed, they can be used by automation engineers, SMEs, or even business users
Release: The following artifacts are delivered in this phase (the list may vary based on the project's need)
Framework components / scripts
Libraries, functions, and data pools
Release notes
Mapping document (traceability matrix)
Design/architecture document
Maintenance: In this phase, we will perform the following:
Here we analyze the failures due to scripts or application issues, logs, and fix them
We enhance the scripts as per the requirement for each cycle/release
The scripts for each cycle are executed
Over a period of time, objects in the GUI and the flow in AUT changes; thus, it is required to change test objects in the Object Repository (OR) and scripts.
We now need to identify the key requirements, which should cover the depth and breadth of the technology required both at the organizational and project level.
The requirements for the test automation are gathered from the following:
Organizational level requirements for test automation
What technology is used in the organization
How tools support the automation of these technologies
Project level or project-specific requirements
Key technical areas
The key steps to conducting a Proof Of Concept (POC) are as follows:
Identify the minimal test set for POC
Verify the technical feasibility
Verify the implementation feasibility if we implement the tools with ease
Find out if there is any known limitation
Submit the POC Summary Report
The key steps to evaluating the tools are as follows:
Identify the key attributes that are most suitable for your project: for example, technology support, technology version, and so on
Give appropriate weight and value to the attributes; the total score is the sum of the product of weight and value as shown:
Total score = ∑ Attribute weight * Attribute value
The following is a comprehensive-but-not-complete list of attributes that we should look into while considering tool selection:
Reliability: This is the ability of a tool to perform and maintain its functions in routine circumstances, as well as hostile or unexpected circumstances.
Capacity: This refers to the possible output of a system, that is, to create and execute scripts in an efficient and effective way.
Operability: This is the ability to work under various configurations and environments.
Performance: The accomplishment of a given task is measured against the presently-known standards of accuracy, completeness, throughput, and speed.
Compatibility: This checks whether the tools support various versions of the technology, browsers, or operating systems.
Non-intrusiveness: The key non-intrusiveness features we should look into for tool selection are as follows:
Platform-/OS-independent
Data-driven capability
Customizable reporting
E-mail notifications
Easy debugging and logging
Support for version control
Extensible and customizable (open APIs should be able to integrate with other tools)
Common driver for logging
Support-distributed execution environment
Distributed application support
License and support (cost)
From a broader perspective, test automation is driven by the following:
The processes defined within an organization
Design and architecture
Automation goals
Best practices for test automation
All of these complement each other for the success of test automation.
A process can be described as a structured approach to perform various tasks and activities that take place during the automation life cycle. Well-defined processes that are laid out by competent organizations provide the guidance to approach automation, from conception to culmination.
In the absence of well-laid processes and historical data it is difficult to estimate, plan, and implement the test automation. In case an organization is implementing test automation for the first time, it becomes the learning curve for future test automations and provides historical data for future projects.
Well-defined architecture/design helps reduce the overheads and conflicts; it also helps the team to collaborate better and achieve better implementation. The key aspect for designing the solution is to provide abstraction from complexities, encapsulation, and technical challenges. We need to develop support libraries for reusability and expandability of the automation solution.
We have to document the design and architecture. The comments in scripts, script headers, and function headers improve the readability and understandability. Documentation helps in enhancing and maintaining the scripts quickly and easily.
A goal is the desired result that a person envisions. It is the plan and commitment to achieve the desired end point in some sort of assumed development. A goal is the key driving and guiding force for the success of test automation. The design requires breaking the framework solution into components and layers. The automation goals should be met at each level or layer for the overall success of test automation.
Note
The objective of defining goals is to get clarity on the outcome. The goals are defined at each layer and have some cascading effect on each other. They also drive and contribute to each other's success.
The overall automation design goals are derived by organizational goals, domains, businesses, and so on. The list may vary from organization to organization. A few of them are as follows:
Increase reusability
Enhance test coverage
Speed up testing for multiple and frequent releases
Ensure consistency
Improve the reliability of testing
Scalability
The following is the list of goals for framework design based on the organization, domain, and so on. This list may vary.
Maintainability
Readability
Extensibility
Reusability
Application independent
Should have short driver script/master script
The following is a list of goal design components of the test automation solution or framework based on the organization, domain, and so on:
The design should be easy to expand and maintain
It should provide abstraction from complexities
Identification of the common functions used across scripts
Decouple complex business functions with utility functions
Decouple test data and test scripts
Creation of robust functions
Ensure that the scripts are executed without human intervention, even in erroneous conditions
The following list of goals for the scripts/code of the test automation is based on the organization, domain, and so on. This list may vary.
The test should always have a common start point and end point
It should clean up the resources to be reused, for example, deleting old log files, result files, and so on
The test should reveal maximum information in case an error occurs
Configure the values rather than hardcoding
Appropriate comments and script headers should be present
The code should be readable and appropriate documentation should be present
The script should be maintainable and easy to modify
Logging utilities should be available
It is essential to set clear, measurable goals for the test automation project. These goals should clearly be brought out. A few examples are:
The number or percentage of test cases to be automated
Enhancing the test coverage by percentage or numbers
Reducing the time to release the build by percentage or numbers
Reduction of the test cycle time of new releases
It is difficult to get the desired outcome or even the ROI without setting measurable goals; this will lead to a high maintenance cost and the changes to the scripts will cause side effects.
Test automation is an integral part of any software development and sustenance project. Some of the best practices that follow will make the test automation project successful.
A misfit tool for test automation increases the effort in designing, creating scripts, and maintaining them, thus increasing the cost of test automation or even failing to achieve the automation goals. Selecting the right tool is winning half the battle. An appropriate tool helps in speeding up script creation, modification, identifying the issues in scripts, and resolving them quickly; in a nutshell, it makes life much easier for the test automation team.
Treat test automation with the same discipline and objectivity as a development project. A successful testing automation project takes a framework-driven approach. Decompose the scripts based on the AUT functionality and divide them into reusable functions. Create a well-defined structure that allows uniform sharing of resources. Prepare appropriate documentation: one that enhances the readability and maintainability. Ensure that the scripting/coding standards have been followed. Finally, you need to test and debug the scripts or code and make sure that the scripts are free from bugs. It is also good to check whether the scripts and resources use version control tools or mechanisms.
It is difficult to extend and maintain the test automation if the framework-driven approach is not used. The overall design becomes chaotic or cumbersome to modify or enhance. The test automation framework allows us to define the scripting methodology to automate tests, create support libraries and common resources to execute the tests, and report the results. Framework allows adding the test cases with minimal effort. It also allows the establishing of a common standard and consistent way to achieve goals for the automation team.
If data is outdated and irrelevant, the test automation suite might not be able to catch bugs. The near-production data (the data which is most similar to production data) ensures that the test data is mapped to the correct fields in the GUI. It should not contain leading and trailing spaces. Use test data with the correct spelling, since it may be relevant for selecting values from the drop-down menu, for example choosing data wisely for reuse across multiple tests. Test data is helpful in enhancing the code coverage. After a certain period the test data might become irrelevant; thus, it should be replaced by relevant and up-to-date test data to find bugs quickly and easily.
Apart from the right tools and a well-laid process, a testing automation project requires a dedicated and skilled team to understand the complexity of the project and translate it into scripts. The gaps in the skillset imbalances the work and delivery, leading to patchwork and low morale of the team members. The leads or managers should see that the team gets adequate training to bridge the gaps (if any) in the skillset.
Without formal reviews, it is likely that defects are not discovered and standards are not followed, leading to potential issues in the integration of framework components. This results in additional cost and effort. A review helps to proactively find issues and potential risks early and mitigate them quickly. Reviews also ensure consistency in the projects and keep the issues in check.
The automation life cycle is a structured approach to design, execute, and maintain test automation. This structured approach is necessary to help steer the test team away from common test program/automation mistakes.
This chapter covered all the aspects of test automation in a broader perspective. Having knowledge of the automation life cycle and key activities in each phase helps in streamlining tasks and goals. Well-defined processes and historical data are helpful in test estimation, and tool evaluation and selection. The architecture or design is based on implementing the test automation and achieving goals. In the next chapter, we will discuss the various features of QTP and concepts that are essential to create reusable scripts.