Software testing in Visual Studio Team System 2008
Before going into the details of the actual testing using Visual Studio 2008, we need to understand the different tools provided by Visual Studio Team System (VSTS) and their usage. Once we understand the tools usage, then we should be able to perform different types of testing using VSTS. As we go along creating a number of different tests, we will encounter difficulty in managing the test similar to the code and its different versions during application development. There are different features such as the Test List Editor and Test View and the Team Foundation Server (TFS) for managing and maintaining all the tests created using VSTS. Using this Test List Editor, we can group similar tests, create number of lists, add, or delete tests from the list.
The other aspect of this article is to see the different file types getting created in Visual Studio during testing. Most of these files are in XML format, which get created automatically whenever the corresponding test is created.
The tools such as the Team Explorer, Code Coverage, Test View, and Test Results are not new to Visual Studio 2008 but actually available since Visual Studio 2005. While we go through the windows and their purposes, we can check the IDE and the tools integration into Visual Studio 2008.
Testing as part of Software Development Life Cycle
The main objective of testing is to find the defects early in the SDLC. If the defect is found early, then the cost will be less, but if the defect is found during production or implementation stage, then the cost will be higher. Moreover, testing is carried out to assure the quality and reliability of the software. In order to find the defect earlier, the testing activities should start early, that is, in the Requirement phase of SDLC and continue till the end of SDLC.
In the Coding phase, various testing activities takes place. Based on the design, the developers start coding the modules. Static and dynamic testing is carried out by the developers. Code reviews and code walkthroughs are also conducted.
Once the coding is completed, then comes the Validation phase, where different phases or forms of testing are performed.
- Unit Testing: This is the first stage of testing in SDLC. This is performed by the developer to check whether the developed code meets the stated requirements. If there are any defects, the developer logs them against the code and fixes the code. The code is retested and then moved to the testers after confirming the code without any defects for the piece of functionality. This phase identifies a lot of defects and also reduces the cost and time involved in testing the application and fixing the code.
- Integration Testing: This testing is carried out between two or more modules or functions together with the intent of finding interface defects between them. This testing is completed as a part of unit or functional testing, and sometimes becomes its own standalone test phase. On a larger level, integration testing can involve putting together groups of modules and functions with the goal of completing and verifying that the system meets the system requirements. Defects found are logged and later fixed by the developers. There are different ways of integration testing such as top-down testing and bottom-up testing:
- The Top-Down approach is followed to test the highest level of components and integrate first to test the high-level logic and the flow. The low-level components are tested later.
- The Bottom-Up approach is the exact opposite of the top-down approach. In this case, the low-level functionalities are tested and integrated first and then the high-level functionalities are tested. The disadvantage in this approach is that the high-level or the most complex functionalities are tested later.
- The Umbrella approach uses both the top-down and bottom-up patterns. The inputs for functions are integrated in the bottom-up approach and then the outputs for the functions are integrated in the top-down approach.
- System Testing: It compares the system specifications against the actual system. The system test design is derived from the system design documents and is used in this phase. Sometimes, system testing is automated using testing tools. Once all the modules are integrated, several errors may arise. Testing done at this stage is called system testing. Defects found in this testing are logged and fixed by the developers.
- Regression Testing: This is not mentioned in the testing phase, but is carried out once the defects are fixed by the developers. The main objective of this type of testing is to determine if bug fixes have been successful and have not created any new problems. Also, this type of testing is done to ensure that no degradation of baseline functionality has occurred and to check if any new functionality was introduced in the software.
Types of testing
Visual Studio provides a range of testing types and tools for software applications. Following are some of those types:
- Unit test
- Manual test
- Web test
- Load test
- Stress test
- Performance test
- Capacity Planning test
- Generic test
- Ordered test
In addition to these types, there are additional tools provided to manage, order the listing, and execute tests created in Visual Studio. Some of these are the Test View, Test List Editor, and the Test Results window. We will look at these testing tools and the supporting tools for managing the testing in Visual Studio 2008 in detail later.
As soon as the developer finishes the code, the developer wants to know if it is producing the expected result before getting into any more detailed testing or handing over the component to the tester. The type of testing performed by the developers to test their own code is called Unit testing. Visual Studio has great support for Unit testing.
The main goal of the unit testing is to isolate each piece of the code or individual functionality and test if the method is returning the expected result for different set of parameter values. It is extremely important to run unit tests to catch the defects in the early stage.
The methods generated by the automated unit testing tool call the methods in the classes from the source code and test the output of each of the methods by comparing them with the expected values. The unit test tool produces a separate set of test code for the source. Using the test code we can pass the parameter values to the method and test the value returned by the method, and then compare them with the expected result.
Unit testing code can be easily created by using the code generation feature, which creates the testing source code for the source application code. The generated unit testing code will contain several attributes to identify the Test Class, Test Method, and Test Project. These attributes are assigned when the unit test code gets generated from the original source code. Then using this code, the developer has to change the values and assert methods to compare the expected result from these methods.
The Unit test class is similar to the other classes in any other project. The good thing here is that we can create new test classes by inheriting the base test class. The base test class will contain the common or reusable testing methods. This is the new Unit testing feature which helps us reduce the code and reuse the existing test classes.
Whenever any code change occurs, it is easy to figure out the fault with the help of Unit tests, rerun those tests, and check whether the code is giving the intended output. This is to verify the code change the developer has made and to confirm that it is not affecting the other parts of the application.
All the methods and classes generated for the automated unit testing are inherited from the namespace Microsoft.VisualStudio.TestTools.UnitTesting.
Manual testing is the oldest and the simplest type of testing, but yet very crucial for software testing. It requires a tester to run all the tests without any automation tool. It helps us to validate whether the application meets various standards defined for effective and efficient accessibility and usage.
Manual testing comes to play in the following scenarios:
- There is not enough budget for automation.
- The tests are more complicated, or are too difficult to be converted into automated tests.
- The tests are going to be executed only once.
- There is not enough time to automate the tests.
- Automated tests would be time-consuming to create and run.
Manual tests can be created either using a Word document or Text format in Visual Studio 2008. This is a form of describing the test steps that should be performed by the tester. The step should also mention the expected result out of testing the step.
Web tests are used for testing the functionality of the web page, web application, web site, web services, and a combination of all these. Web tests can be created by recording the interactions that are performed in the browser. These can be played back to test the web application. Web tests are normally a series of HTTP requests (GET/POST).
There are different validation rules and extraction rules used in web testing. The validation rules are used for validating the form field names, texts, and tags in the requested web page. We can validate the results or values against the expected result as per business needs. These validation rules are also used for checking the time taken for the HTTP request.
At some point in time, we need to extract the data returned by the web pages. We may need the data for future use, or we may have to collect the data for testing purposes. In this case, we have to use the extraction rules for extracting the data returned by the page requested. Using this process, we can extract the form fields, texts, or values in the web page and store it in the web test context or collection.
Web tests cannot be performed only with the existence of a web page. We need some data to be populated from the database or some other source to test the web page functionality and performance. There is a data binding mechanism used in Web test, which is used for providing the data required for the requested page. We can bind the data from a database or any other data source. For example, the web page would be a reporting page that might require some query string parameters as well as the data to be shown in the page according to the parameters passed. To provide data for all these data-driven testing, we have to use the concept of data binding with the data source.
Web tests can be classified into Simple Web tests and Coded Web tests. Both these are supported by VSTS.
- Simple Web tests are very simple to create and execute. It executes on its own as per the recording. Once the test is started, there won't be any intervention. The disadvantage is that it is not conditional. It's a series of valid flow of events.
- Coded Web tests are bit more complex, but provide a lot of flexibility. For example, if we need some conditional execution of tests based on some values then we have to depend on this coded web test. These tests are created using either C# or Visual Basic code. Using the generated code we can control the flow of test events. But the disadvantage is its high complexity and maintenance cost.
Load testing is a method of testing used in different types of testing. The important thing with Load testing is that it is about performance. This type of testing is conducted with other types of testing, which means that it can be performed along with either Web testing or Unit testing.
The main purpose of load testing is to identify the performance of application based on different scenarios. Most of the time, we can predict the performance of the application that we develop, if it is running on one machine or a desktop. But in the case of web applications such as online ordering systems, we know the estimated maximum number of users, but do not know the connection speeds and location from where the users will access the web site. For such scenarios, the web application should support all the end users with good performance irrespective of the system they use, their Internet connection, the place, and the tool they use to access the web site.
So before we release this web site to the customers or the end users, we should check the performance of the application so that it can support the mass end user group. This is where load testing will be very useful in testing the application along with Web test or Unit test.
When a Web test is added to a Load test, it will simulate multiple users opening simultaneous connections to the same web application and making multiple HTTP requests. Load testing in Visual Studio comes with lots of properties which can be set to test the web application with different browsers, different user profiles, light loads, and heavy loads. Results of different tests can be saved in a repository to compare the set of results and improve their performance.
In case of client server and multi-tier applications, we will be having a lot of components which will reside in the server and serve the client requests. To get the performance of these components, we have to make use of a Load test with a set of Unit tests. One good example would be to test the data access service component that calls a stored procedure in the backend database and returns the results to the application that is using this service.
Load tests can be run either from the local machine or by submitting to a rig, which is a group of computers used for simulating the tests remotely. A rig consists of a single controller and one or more agents.
Load tests can be used in different scenarios of testing:
- Stress testing: This checks the functionality of the application under heavy load. The resource provided to the application could vary based on the input file size or the size of the data set, for example, uploading a file which is more than 50MB in size.
- Smoke testing: This checks if the application performs well for a short duration with a light load.
- Performance testing: This checks the responsiveness and throughput of the application with different loads.
- Capacity Planning test: This checks the application performance with various capacities.
As we know, there are different types of testing required to build quality software. We take care of running all these tests for the applications we develop. But we also have an order in which to execute all these different tests. For example, we do the unit testing first, then the integration test, then the smoke test, and then we go for the functional test. We can order the execution of these tests using Visual Studio.
Another example would be to test the configurations for the application before actually testing the functionality of the application. If we don't order the test, we would never know whether the end result is correct or not. Sometimes, the tests will not go through successfully if the tests are not run in order.
Ordering of tests is done using the Test View window in Visual Studio. We can list all the available tests in the Test View and choose the tests in the same order using different options provided by Visual Studio and then run the tests. Visual Studio will take care of running the tests in the same order we have chosen in the list.
So once we are able to run the test successfully in an order, we can also expect the same ordering in getting the results. Visual Studio provides the results of all the tests in a single row in the Test Results window. Actually, this single row result will contain the results of all the tests run in the order. We can just double-click the single row result to get the details of each tests run in the ordered test.
Ordered test is the best way of controlling the tests and running the tests in an order.
We have seen different types and ways of testing the applications using VSTS. There are situations where we might end up having other applications for testing, which are not developed using Visual Studio. We might have only the executables or binaries for those applications. But we may not have the supported testing tool for those applications. This is where we need the generic testing method. This is just a way of testing third-party applications using Visual Studio.
Generic tests are used to wrap the existing tests. Once the wrapping is done, then it is just another test in VSTS.
Using Visual Studio, we can collect the test results, and gather the code coverage data too. We can manage and run the generic tests in Visual Studio just like the others tests.
Lists of tests
This is the first node in the list which is initially empty. It also asks us to create the list and add the tests to the list.
We can click on the here link and give the details for the new test list in the window that opens up as Create New Test List. Let's call that test list Manual Tests to group all the available manual tests.
Now in the left pane of the Test List Editor, select the second option Tests Not in a List. Do not check the option but just select the option. This will display all the available tests in the project, which are not part of any other Test list as shown below:
Now choose both the Manual tests on the right pane and just drag and drop them on the Manual Tests node on the left pane, which we created earlier. The other way of doing the same is, after selecting both manual tests, by right-clicking and choosing the New Test List... option. We will get a window similar to what we saw earlier in Create New Test List.
Since we have already created the Test list, we will just drag and drop the tests on the test list name. Now when we select the newly created Manual Tests list, we can both the manual tests listed on the right-side pane.
Now if we select the node Tests Not in a List, we can see only the other tests that we had not selected in our previous steps to add to the Manual Tests list.
Select the third node in the left pane All Loaded Tests. This option will list down all the tests available, whether it is added to a Test List or not. We can also see the Test List name to which a particular test is added; and it will remain empty for the tests which are not part of any of the Test Lists. The following screenshot shows all the loaded tests in the project:
After creating the Test List, we can run the tests based on the list we have created. The purpose of creating the Test List is to group the tests together so that we can run them all together by selecting the Test List using the checkbox option to the left of the test list name and then selecting the Run or Debug option in the toolbar. Each of the nodes in the Test List Editor has checkboxes. Using these checkboxes, we can select the individual tests, a Test List, multiple tests within the Test list, and tests that are not in a list. Then we can run the tests using the Run Checked Tests or Debug Checked Tests option in the toolbar.
There are two more options in editor toolbar: Load Metadata and Import Metadata. These are helpful in loading the existing Test Lists, which are already saved but not available in the current list. This Import and Load refer to the metadata file that gets created for the Test Project under the solution. It's the file with the extension vsmdi, which contains all the information about the tests lists.
All other options in the toolbar are the same as the options available in the Test View window such as grouping, filtering, and adding or removing columns.
This window shows the results of the tests run. It shows the error messages or the statuses of the multiple tests run. We can also reselect the tests from the Test Results window and rerun the tests. Using this window, we can export the test results or import the results already exported. This is useful in viewing the result in the Test Results window and rerun the test.
Now, as we have seen in the previous section about the Test List Editor, select the Manual Tests list and run the tests.
Now, we will consider the test, manualtestdoc as passed, and the test ManualTestText as failed. While running the tests, we can find the tests in progress and the status of the completed tests in the Test Results window as shown below. It's just the starting of the tests. So both the tests are in pending stage.
After completing both the tests, we can see the results as shown in the following screenshot. It will clearly indicate the status for each result. The window also shows the summary of the results, passed and failed. If we want to rerun the test, we don't need to go to the List Editor. From the results window, we can select the tests and rerun them. Using the checkbox before the test in the list window, we can select the test and then choose the Run or Debug option to rerun it.
In the preceding screenshot, we can also notice that there are two panes similar to the Test List Editor. The second option in the toolbar shows the list of Test Lists available and their corresponding tests in the right pane. It is the same option that we saw in the Test List Editor.
From the test result, we can also get the details of the test list run. We can get the details such as start time, end time, test location, and the user who ran the test and also the summary of the test results as shown in the following screenshot:
There is another option in the Test Results, which is the option to export the results. Whenever we finish our testing, we always would want to save the results of the tests for our further analysis. In the toolbar, we can see the option for exporting the test results. It provides two options, Export Test Run Results... to export all test results and Export Selected Test Results... to export only the selected test run result. We can also import the saved result using this window.
Once we choose any of these options, we can see the file dialog for saving the file. We can also notice that the file type is *.trx, which is the Visual Studio Test Results file for saving all the test results.
The .trx test result file is created by the VSTS on execution of the test every time. The file is named after the user, machine name, the date and time of execution. In the previous image, you can see the test name as Subashni@MyPC 2008-02-16 19:59:50, which shows the name of the user logged in and running the test, the machine name, and then the date and time of execution. The .trx file is an XML file with all information about the test result. If you open the .trx file, it will open the test result window in Visual Studio and show the details in the Test Results window. The contents of the test result file would look like the code shown below. This is just a sample and part of the generated test result file:
<?xml version="1.0" encoding="UTF-8"?>
name="Subashni@MyPC (2008-02-16 29:59:50)" runUser="Subashni"
<Description>Test configuration file with the user defined
<Timeouts runTimeout="1800000" testTimeout="2700000" />
<Remote controllerName="<Local - No controller>" />
TestResultsSubashni_MyPC _2008-02-16 19_59_50_">
<DeploymentItem filename="Test.dll" />
<DeploymentItem filename="C:Program Files
Microsoft Visual Studio 9.0Common7IDE
<DeploymentItem filename="ClassLibrary1.dll" />
<NamingScheme baseName="Subashni@MyPerPC" useDefault="false" />
Code coverage results
Code coverage provides information about codes and how many kinds of codes are covered during the test run. This is to view the statistical information about the test run. This result will show how many lines of code are executed and how many lines are not executed. We can also get the color coding for the lines run, and different color coding for the lines not run. This is also a kind of test result.
Before starting the code coverage we have to configure the code coverage. Select the Test | Edit Test Run Configurations | Local Test Run option. These configurations are stored in the configuration file, localtestrun.testrunconfig. In the configuration window, select the option Code Coverage in the left pane. On the right-side pane, we can see the list of all the available artifacts for code coverage. The artifacts could be an application, or a library, or any type of assembly. We can choose the artifacts from the list, or we can also import the artifacts using the Add Assembly option and then choose that for the code coverage. As shown in the following screenshot, let us select the test project library, TestProjectforTestingApp.dll.
When we click on apply in the above window, the code coverage is enabled only for the assemblies selected. When the application is executed, we can see the code coverage window collecting information about the source code covered in testing as shown in the following screenshot. The code coverage window is available under the menu option Test | Windows | Code Coverage Results. Lets try running the unit test application available in the test list editor under the Tests Not in a List which has the Unit test. After running the test, we can see the code coverage results as shown in the following screenshot:
The Code Coverage Results window shows each block covered, blocks not covered, and the percentage of coverage for blocks. This applies to each run of the test. Next time we run the same test, we can see a different set of results under a new name.
In the Unittest1 class, we can also notice the color coding based on the coverage. This is the code which is covered under testing, and code that is not covered will be highlighted with color coding.
After completing the test, we can also export the coverage results similar to the test results. We can also import the coverage results which were already exported. There is another new option, Merge Results. This window will list all the coverage results from which we can select the results and merge them to get the end result of the code coverage collectively from various results.
Overview of XML files for testing tools in Visual Studio 2008
There are several XML files used by the VSTS Test Edition for storing the test information. All these documents should conform to the schema defined in the TestTypes.xsd file.
We have seen some of the XML document types in the previous examples, and here is the summary of all the XML documents for different test types supported by Visual Studio.
- Test Metadata file: This file stores information about all the tests in the solution. It has the extension .vsmdi.
- Test Results file: This file contains the results of the tests that were run. This file will be created automatically by Visual Studio when any test is run. It has the extension .trx.
- Test Run Configuration file: This file contains the settings that specify how the test runs should proceed further. It has the extension .testrunconfig.
- There are other XML files for different test types with some specific extensions such as .webtest, .loadtest, .orderedtest, .generictest.
XML schema was changed from Visual Studio 2005 Team Edition to the release of Visual Studio Team System 2008 Test Edition. So the XML documents that were created using Visual Studio 2005 Team Edition for Testers need to be upgraded to VSTS 2008 Test Edition before starting to use those XML documents. These XML documents can be converted to the current format using the exe, VSTST-FileConverter.exe.
This file accepts one parameter which is the XML document for the test created using Visual Studio 2005. For example, if the test is an ordered test with the name SampleOrderedTest1, then it should be converted using the converter to:
To get help on the file converter, we can use the following command: