Software testing is one of the most important phases of the software development life cycle (SDLC). Delivery of the software product is based on following good SDLC practices of analysis, design, coding, testing, and by all means meeting the customer requirements. The quality of the product is measured by verifying and validating the product based on the defined functional and non-functional requirements for product. The testing tools and techniques play an important role in simulating the real-life scenarios and the user load required for verifying the stability and reliability of the product. For example, testing a web application with 1,000 concurrent users is a very time consuming and tedious task, if we do it manually considering the required resources. But the testing tools that are part of Visual Studio 2012 can simulate such scenarios and test it with limited resources and without manual intervention during testing. Visual Studio 2012 provides tools to conduct different types of testing, such as Unit testing, Load testing, Web testing, Ordered testing, Generic testing, and Exploratory testing.
This chapter covers the following topics and provides a high-level overview of the testing tools and techniques supported by Visual Studio 2012:
Testing as part of the software development life cycle
Types of testing
Test management in Visual Studio 2012
Testing tools in Visual Studio 2012
Before getting into the details of how to perform testing using Visual Studio 2012, let us familiarize different tools provided by Visual Studio 2012 and its usage. Visual Studio provides tools for testing as well as test management such as the Test List Editor and the Test View. The Test Projects and the actual test files are maintained in Team Foundation Server (TFS) for managing the version control of the source and history of changes.
The other aspect of this chapter is exploring the different file types generated in Visual Studio during testing. Most of these files are in the XML format, which are created automatically whenever a new test is created.
For readers new to Visual Studio, there is a brief overview on each window we are going to deal with throughout all or most of the chapters in this book. While we go through the windows and their purposes, we can check the Integrated Development Environment (IDE) and the tools integration with Visual Studio 2012.
Microsoft Visual Studio 2012 has different editions tailored to the needs. You need to have the respective edition as prerequisite to use any of the testing features explained in this book. The following table shows supported edition of Visual Studio 2012 for the testing features.
|
Testing features |
Ultimate with MSDN |
Premium with MSDN |
Test Professional with MSDN |
Professional with MSDN |
Professional |
|---|---|---|---|---|---|
|
Yes |
Yes |
Yes |
Yes | ||
|
Yes |
Yes | ||||
|
Yes |
Yes | ||||
|
Yes |
Yes |
Yes | |||
|
Yes |
Yes |
Yes | |||
|
Yes |
Yes |
Yes | |||
|
Yes | |||||
|
Yes |
Yes |
Yes |
Microsoft Test Manager 2012 (MTM) is a standalone product from Microsoft, which integrates with Team Foundation Server for test management. MTM is used in creating and managing multiple Test Plans, cloning Test Plans, creating Test Suites, creating manual test steps and test cases, and maintaining the same. MTM also provides various reports for Test Plan results. In 2012 version, MTM has the new feature of exploratory testing, maintaining records, and test steps during exploratory testing.
Lab environments can be created in MTM using the controller and agents. This is required when running load tests with multiple agents.
The main objective of testing is to find the early defects in the SDLC. If the defect is found early, then the cost will be lower than when the defect is found during the production or in the implementation stages. Moreover, testing is carried out to assure the quality and reliability of the software. In order to find the defect as soon as possible, the testing activities should start early, that is, in the Requirement phase of SDLC and continues till the end of the SDLC. The testing team should create the test cases based on the defined requirements.
The Coding phase of the SDLC includes various testing activities to validate and verify the functionality based on the design and the developer's code for the design. The developers themselves conduct the tests. In case of Test driven development, the test scripts and test scenarios are created first based on the requirement and the code is developed.
As soon as the developer completes the coding, the developer conducts the unit testing
Unit testing: This is the first level of testing in the SDLC. The developer takes the smallest piece or unit of testable code and determines whether the code behaves exactly as expected. In object-oriented programming, the smallest unit is a method which belongs to a class. The method usually has one or few inputs and one output. Frameworks, drivers, Stubs and mock, or fake objects are used to assist in unit testing.
Once the coding is complete for the agreed requirements, all the units are integrated and the product is built as a single package. Then the other phases or forms of testing are executed.
Integration testing: This type of testing is carried out between two or more modules or functions along with the intent of finding interface defects between them. This testing is completed as a part of unit or functional testing, and sometimes becomes its own standalone test phase. On a larger level, integration testing can involve putting together groups of modules and functions with the goal of completing and verifying that the system meets the system requirements. Defects found are logged and fixed later by the developers. There are different ways of integration testing such as top-down and bottom-up , which are as follows:
Top-down approach: This is the incremental testing technique which begins with the top level modules followed by low-level modules. The top-down approach helps in early detection of design errors which helps in saving development cost and time as the design errors can be fixed before implementation.
Bottom-up approach: This is exact opposite to the top-down approach. In this case the low level functionalities are tested and integrated first and then followed by the high level functionalities.
Umbrella approach: This approach uses both the top-down and bottom-up patterns. The inputs for functions are integrated in bottom-up approach and then the outputs for functions are integrated in the top-down approach.
System testing: This type of testing is used for comparing or verifying the specifications against the developed system. The system test design is derived from the design documents and is used in this phase for planning and executing the tests. System testing is conducted after all the modules are integrated and completed with Integration testing. To avoid repeating the same process during multiple cycles of system testing, the tests are automated using automation testing tools. Once all the modules are integrated, several errors may arise because of dependencies and various other factors. The defects are usually maintained using a defect tracking tool and the development team prioritizes and fixes the defects. There are different types of testing followed under system testing, but they differ from organization to organization. Here are the common types of tests widely followed in the industry:
Sanity testing: Whenever there are some defect fixes to the existing product and because of that a new build is created, sanity test is conducted on that build instead of performing full testing on the software. Sanity test is conducted to make sure that the existing functionality of the product is not impacted or broken because of the defect fixes.
Regression testing: The main objective of this type is to determine if defect fixes or any other changes have been successful and have not introduced any new defects. This is also to verify if the existing functionalities are not affected.
Visual Studio provides a range of testing types and tools for testing software applications. The following are some of those types:
Unit test
Manual test
Exploratory test
Web test
Coded UI test
Load test
Ordered test
Generic test
The unit testing tool is integrated along with Visual Studio and developers can use any of the Visual Studio supported language to write the unit testing. The manual test and exploratory test can be used during regression and is integrated with the Test Manager tool to track the test cases and defects when the test is conducted. Web Test and Coded UI Test in Visual Studio is used for system testing to record and playback the test steps. The load test tool is used during system testing cycle for testing performance and stability of the application with user load, and is integrated with Test Manager. The Generic test is again a part of the system testing to test the third-party components and the ordered test is to enable the testing order during Test Runs.
For all of the above testing types, Visual Studio provides tools to manage, order the listing, and execute tests. The next few sections provide details of these testing tools and the supporting tools for managing testing in Visual Studio 2012.
Unit testing is one of the earliest phases of testing the application. In this phase the developers have to make sure that the unit of testable code delivers the expected output. It is extremely important to run unit tests to catch defects in the early stage of the software development cycle. The main goal of the unit testing is to isolate each piece of the code or individual functionality and test if individual method is returning the expected result for different sets of parameter values.
A unit test is a functional class method test by calling a method with the appropriate parameters, exercises it and compares the results with the expected outcome to ensure the correctness of the implemented code. Visual Studio 2012 has great support for unit testing through the integrated automated unit test framework, which enables developers to create and execute unit tests.
Visual Studio generates the test methods and the base code for the test methods. It is the responsibility of the developer to modify the generated test methods and customize the code for actual testing. The code file contains several attributes to identify the Test Class, Test Method, and Test Project. These attributes are assigned when the unit test code is created for the original source code. Here is the sample of the unit test code:
Once a unit test is created for a testable unit of code, the developers can use it with multiple combinations of input parameters to make sure the actual result is as per the expected result.
All the methods and classes generated for the unit testing are inherited from the Microsoft.VisualStudio.TestTools.UnitTesting namespace. This namespace is only used when the default Visual Studio integrated testing tool is used. This namespace contains many classes and attributes to provide enough information for the test engine to determine data source, test execution, execution order, deployment, and results.
Visual Studio also provides the flexibility to integrate unit testing tools such as Unit and XUnit for which the adapters need to be installed. After installing the tool, the respective namespaces can be used for generating calls and unit testing methods.
Manual testing is the oldest and simplest type of testing, but yet very crucial for software testing. The tester would be writing the test cases based on the functional and non-functional requirements and then test the application based on each written test case. It helps us to validate whether the application meets various standards defined for effective and efficient accessibility and usage.
Manual testing can be an alternative in the following scenarios:
The tests are more complex or too difficult to convert into automated tests.
There is not enough time to automate the tests.
Automated tests would be time consuming to create and run.
There are not enough skilled resources to automate the tests.
The tested code hasn't stabilized sufficiently for cost effective automation.
We can create manual tests by using Visual Studio 2012 very easily. A very important step in manual testing is to document all the test steps required for the scenario with supporting information in a separate file. Once all the test cases are created, we should add the test cases to the Test Plan in order to run the test and gather the Test Result every time we run the test. The Microsoft Test Manager tool helps us in adding or editing the test cases to the Test Plan. The manual testing features supported by Visual Studio 2012 are as follows:
Running the manual test multiple times with different data by changing parameters.
Create multiple test cases using an existing test case and then customize or modify the test.
Sharing test steps between multiple test cases.
Remove the test cases from the test if no longer required.
Adding or copying test steps from Microsoft Excel or Microsoft Word or from any other supported tool.
Including multiple lines and rich text in manual test steps.
There are a lot of other manual testing features supported in Visual Studio 2012. We will see those features explained in Chapter 2, Test Plan, Test Suite, and Manual Testing.
Exploratory testing is an open approach to testing without any process and test cases. The only known fact is the user story. The objective of this testing is to test the existing application or feature, and to find any improvements required, defects, broken links, and familiarize with the existing system. This type of testing has been followed for many years, but there was no tool to support the testing and capture the defects and steps. It was a tedious process to document the steps and capture supporting screenshots.
The Microsoft Test Manager (MTM) has the new feature to perform the exploratory testing and capture the screenshots, test steps, test case, comments, attachments, and defects automatically. The testing actions are stored as test cases so that it is easy while retesting.
To start exploratory testing, open the MTM and navigate to Testing Center | Test |Do Exploratory Testing. Now by selecting a work item requirement and then clicking on Explore work item will associate the recording of the test with the work item. Any test cases or defects created during Exploratory session will automatically get linked to the work item. The following screenshot shows a sample Exploratory testing session started for a work item:
During Exploratory testing, all actions performed on the screen are recorded except the actions performed in MTM and Office applications. To change this setting, configure the settings in the Test Plan properties.
A detailed walk-through the Exploratory testing is covered in Chapter 12, Exploratory Testing and Reporting which talks about Exploratory testing and reporting.
Web performance tests are used for testing the functionality and performance of the web page, web application, website, web services, and a combination of all of these. Web tests can be created by recording the HTTP requests and events during user interaction with the web application. The recording also captures the web page redirects, validations, view state information, authentication, and all the other activities. All these are possible through manually building the web tests using Web test. Visual Studio 2012 provides Web performance test features, which capture all HTTP requests and events while recording user interaction and generating the test.
There are different validation rules and extraction rules used in Web performance tests. The validation rules are used for validating the form field names, texts, and tags in the requested web page. We can validate the results or values against the expected result as per the business needs. These validation rules are also used for checking the processing time taken for the HTTP request.
Extraction rules in Web performance tests are used for collecting data from the web pages during requests and responses. The collection of these data will help us in testing the functionality and expected result from the response.
Providing sufficient data for the test methods is very important for the success of automated testing. Similarly for web tests we need to have a data source from which the data will be populated to the test methods and the web pages will be tested. The data source could be a database or a spread sheet or an XML data source or any other form of data source. There is a data binding mechanism in Web tests which takes care of fetching data from the source and provides the data to the test methods. For example, a reporting page in a web application definitely needs more data to test it successfully. This is also called the data-driven web test.
Web tests can be classified into Simple Web test and Coded Web test. Both of these are supported by Visual Studio.
Simple Web tests: This includes generating and executing the test as per the recording with a valid flow of events. Once the test is started, there won't be any intervention and it won't be conditional.
Coded Web tests: This is more complex, but provides a lot of flexibility. These types of tests are used for conditional execution based on certain values. Coded Web tests can be created manually or generated from a web test recording and languages such as C# or VB.NET can be chosen while generating the code. The generated code can be customized to better control the flow of test events. A coded Web test is a powerful and highly customizable test for the web requests.
Coded UI Tests (CUIT) are the automated way of testing the application user interface. In any UI intensive application, the functionality of the application is verified manually through UI and this happens after the development. Next time there is any change to any of the backend functionality, the application should be retested again. CUIT helps us in saving time spent testing through UI multiple times manually. CUIT Builder helps us in recording the UI test step actions and then generates code out of it. After the test is created, we can modify the code and customize the actions and data values captured during recording.
A Coded UI Test generates several supporting files as part of the testing. The UIMap object represents the controls, windows, and assertions. Using these objects and methods we can perform actions to automate the test. The coded UI Test supporting files are as follows:
CodedUITest.cs: This file contains the test class, test methods, and assertions.UIMap.uitest: This is the XML model forUIMapclass, which contains the windows, controls, properties, methods, and assertions.UIMap.Designer.cs: This contains the code for theUIMap.uitestXML file.UIMap.cs: All customization code for the UI Map would go into this file.
The following screenshot shows the Coded UI Test with the default files created for the test:
Load testing is a method of testing, which is used to identify the performance of the application under maximum workload. In case of a desktop or a standalone application, the user load is predictable, and thus easy to tune the performance, but in case of a multiuser application or a web application, it is required to determine the application behavior under normal and peak load conditions.
Visual Studio provides a load test feature, which helps in creating and executing load test with multiple scenarios. The following are the parameters set using the load test wizard:
Load Test Pattern: This defines the number of users and the user load pattern to be followed during the test.
Test Mix Model: This defines the model to be followed either by number of tests or by number of virtual users, or based on the user pace or by order.
Test Mix: This includes the tests to be part of the load tests.
Browser Mix and Network Mix: These define the possible browsers and the networks to follow while testing.
Counter Sets: This defines the performance counters to collect from the load Test Agents and the system.
Run settings: This defines the duration of the Test Run.
If the application is a public-facing website or one with a huge customer base, then it is better to perform load tests with real or expected scenarios. The Visual Studio load test makes use of the Web test recording or the unit test during load Test Run.
The load test is always driven by the collection of Web and Unit tests. A web test is used to simulate the scenario of concurrent users using the website and making multiple HTTP requests. The load can be configured to start with a minimum number of virtual users and then gradually increase the user count to check the performance at multiple stages of user load until it reaches the peak user load.
A unit test can be included as part of the load test in case of testing the performance of a service or individual method to find out the servicing capacity and threshold for client requests. One good example would be to test the data access service component that calls stored procedure from the backend database and returns the results to the client application.
The load test captures the results of individual tests within the Test Run. This helps us to identify the failed tests and debug and analyze them later. The results of all load tests can be saved in a repository to compare the set of results and then take necessary measures to improve performance.
Visual Studio has the Load test analyzer to provide the summary and details of Test Runs from the load Test Result.
Load testing properties, working with tests, and analyzing the load Test Results are explained in detail later in this book in Chapter 7, Load Testing.
Ordered test is just a container which holds the order in which a sequence of tests should be executed. All required tests should be ready and available to get added to the ordered test. Each test is independent and there is no dependency here. It is just the sequence of execution that is maintained in the ordered tests.
Test execution and results follow the sequence defined in the ordered test. The result of individual test is maintained in the repository. We can check the results anytime and analyze it.
Reordering the tests, adding new tests, and removing an existing test from the order are all possible through the Ordered Test Editor in Visual Studio.
An ordered test is the best way of controlling and running several tests in a defined order.
Generic test is useful in testing an existing executable file. It's the process of wrapping the executable file as a generic test and then executing it. This type of testing is very useful when testing a third party component without the source code. If the executable requires any additional files for testing, the same can be added as deployment files to the generic test. The test can be run using the Test Explorer or a command-line command.
By using Visual Studio, we can collect the Test Results and gather code coverage data too. We can manage and run the generic tests in Visual Studio just like other tests. In fact, the Test Result output can be published to the Team Foundation Server to link it with the code built used for testing.
Visual Studio has great testing features and management tools for testing. These features are greatly improved from previous versions of Visual Studio. The Test Impact View is the new test management tool added to the existing tools, such as Test View, Test List Editor, Test Results, Code Coverage Results, and Test Runs from the main IDE.
Visual Studio provides tools to create, run, debug, and view results of your tests. The following screenshot is the overview of the tools and windows provided by Visual Studio for viewing the test and output details:
Let us create a new Test Project using Visual Studio 2012 and then test a sample project to get to know about the tools and features:
Open Visual Studio 2012 and create a new solution. Let's not get into the details of sample application, AddNumbers, but create the Test Project and look at the features of the tools and windows. The application referred throughout this chapter is a very simple application for adding two numbers and showing the result.
Now in a similar way to adding the projects and code files to the solution, create the Test Project and test files and add the Test Project to the solution.
Select the solution and add a project using the shortcut menu options Add | New Project.... Then select the project type as Test from the list of project types under the language. Next select a template from the list. Visual Studio 2012 has three templates as follows:
For the sample testing application, select the second option, Unit Test Project. This option creates the project and also adds the unit test to the project.
The first option creates a Coded UI Test to capture the UI actions and controls, and automate the testing by generating and customizing the code. The last option is to record the user actions and re-run the recording to test with validations and verification rules, use the recording to test the performance and stability of the system under multiple user load.
The Context menu from the project has the option to choose new tests. The menu provides six different options for creating and adding the tests. The following is the file extension for each of the Visual Studio test types shown in the preceding screenshot:
.vbor.cs: This extension is for all types of Unit Test and Coded UI Test..generictest: This extension is for the Generic Test type..loadtest: This extension is for the Load Test type..webtest: This extension is for the Web Performance Test type.
After selecting the test type, the test file gets created and added to the project with a default name and extension. Open the properties and change the name as required.
The next step is to use the Test Explorer window to view and run the tests that are created.
The Test Explorer window helps us to run tests from multiple projects in a solution. On building the Test Projects, the tests in each project appear in the Test Explorer window. To open Test Explorer, navigate to Test | Windows | Test Explorer. The tests are grouped into four different categories in Test Explorer, such as Failed Tests, Passed Tests, Skipped Tests, and Not Run Tests as shown in the following screenshot:
The Test Explorer window has the following options to run the tests:
Choose Run All... to run all the tests in the solution
Choose Run… and then a group to run all the tests under that group
Select an individual test, open the Context menu, and then select Run Selected Tests to run only the selected tests
To view the details of the Test Run, select the test in the Test Explorer window. The Details pane displays the details as follows:
Source: This is the source file name and the line number of the test method.
Status: This is the test status whether it has passed or failed.
Message: If the test is failed, the detailed message of the failure is also displayed.
Elapsed time: This is the time that the method took to run.
StackTrace: This is the stack trace information for the failed test.
At any time if you double-click on the Test option or select Test and choose Open Test, Visual Studio opens the source code of the selected test. This is very helpful when starting to debug the code.
Visual Studio provides this code coverage feature to find out the percentage of code that is covered by the test execution. Through this window we can find out the number of lines covered by the test in each method.
Select the test from the Test Explorer window, and then right-click on the test and select Analyze Code Coverage for Selected Tests, or you can open the same by navigating to Test | Windows | Code Coverage Results from the Menu option. The following screenshot shows the code coverage results for the selected test. The result window provides information such as number of code blocks not covered by the test, percentage of code blocks not covered, covered code blocks, and the percentage of covered code blocks from the selected assembly:
This is the new standalone product introduced, but this is not a part of Visual Studio 2012 Premium. It is a part of Visual Studio Test Professional and Visual Studio Ultimate. This is the functional testing tool, which provides the ability to create and execute manual tests and collect the results. This tool works without Visual Studio but does require a connection to the Team Foundation Server and the Team Project.
The Testing Center is used for creating Test Plans and creating Test Suites and test cases for the Test Plans. We can also associate the requirements to the Test Plans. The other features such as running the manual test and capturing the Test Results and defects, tracking the Test Results using existing queries and creating custom queries, organizing the test cases and shared steps for test cases, and maintaining the test configurations , that are supported by Testing Center.
Lab Center is used to set up and create lab environments for the test execution. The environments are created by using the Physical and Virtual machines with set of configurations. Later the environment is deployed so that the test would be conducted using the specified environment.
MTM should be connected to the TFS Team Project to create the Test Plans and test cases. The first task in opening MTM is to connect to the TFS Team Project from the Team Project collection , as shown in the following screenshot:
The Test Plan window in the Testing Center allows the creation of new Test Suites, test cases, and adding test cases based on requirements. Any number of test cases can be created or added and configured through this window. The first step is to create the Test Plan and Test Suite. Each Test Plan contains a set of Test Suites which helps us to plan the testing effort. For example, we can create a Test Plan for each sprint, if we are using agile methodology for the software development. The following screenshot has one Test Plan (MytestPlan) with two Test Suites as Add Numbers Test and Second Test Suite:
The next step is to create or add test cases to the suite. The requirements can be added to the plan and then test cases can be associated to the requirements. In the above example, three requirements are added to the first Test Suite, Add Numbers Test and new test cases are associated with the requirements. These requirements are available in TFS under the Team Project. If we follow the Application lifecycle management and tools available in TFS then we should have the requirements created as part of the requirements phase already.
The creation of a test case involves the definition of a lot of properties for it. Each testing step and the expected result should be defined. The user scenarios, links, and attachments can be associated to the test cases. The test case can be classified under the specific area path and iteration path for the Team Project in TFS. Other properties such as Assigned To, State, Priority, and Automation status can be set for the test case as shown in the following screenshot:
The Lab Center in MTM helps us to create and configure different virtual/physical environments for our Test Runs, test settings such as defining the roles and configuring the data and diagnostics information for the selected roles, configuring the Test Controllers required for the test, and configuring the test library to store the environment information. The following screenshot shows the Lab Center without any environment:
We can see the details of these configurations later in Chapter 13, Test and Lab Center which explains the features of Testing Center and Lab Center.
There are lots of new testing features added to Visual Studio 2012 particularly the coded UI testing, manual testing using the Test Manager Standalone tool. The manual testing is very well structured with lot of options and is handled separately using the MTM tool. The MTM tool contains Testing Center and Lab Center, which helps us to maintain the test cases, test configurations, and testing environments required to simulate actual user load to test the application performance. This chapter provides the high-level information on the tools and techniques available and the new techniques added to Visual Studio 2012. Each of these testing techniques is explained in detail in the coming chapters with detailed examples.
The next chapter explains the details of maintaining the test cases by creating Test Plan, Test Suites, and then the test cases themselves. Action recording and creating test cases for manual tests is also covered in detail in the next chapter.
