Ordered and Generic Tests in Visual Studio 2010

A step by step guide to understanding the features and concepts of testing applications using Microsoft Visual Studio 2010 with this book and eBook

 

Software Testing using Visual Studio 2010

Software Testing using Visual Studio 2010

A step by step guide to understand the features and concepts of testing applications using Visual Studio.

  • Master all the new tools and techniques in Visual Studio 2010 and the Team Foundation Server for testing applications
  • Customize reports with Team foundation server.
  • Get to grips with the new Test Manager tool for maintaining Test cases
  • Take full advantage of new Visual Studio features for testing an application's User Interface
  • Packed with real world examples and step by step instructions to get you up and running with application testing

 

        Read more about this book      

Generic Tests are just like any other tests except that they are used for running the existing third party tool or program, which can also be run using command line.

Ordered tests

The following screenshot shows the list of all the tests. You can see that the tests are independent and there is no link between the tests. We have different types of tests like Unit Test, Web Performance Test, and Load Test under the test project. Let's try to create an ordered test and place some of the dependent tests in an order so that the test execution happens in an order without breaking.

Creating an ordered test

There are different ways of creating ordered tests similar to the other tests:

  1. Select the test project from Solution Explorer, right-click and select Add Ordered Test, and then select ordered test from the list of different types of tests. Save the ordered test by choosing the File | Save option.
  2. Select the menu option Test then select New Test..., which opens a dialog with different test types. Select the test type and choose the test project from the Add to Test Project List drop-down and click on OK.

Now the ordered test is created under the test project and the ordered test window is shown to select the existing tests from the test project and set the order.

Software Testing using Visual Studio Team System 2010

The preceding window shows different options for ordering the tests.

The first line is the status bar, which shows the number of tests selected for the ordered test.

The Select test list to view dropdown has the option to choose the display of tests in the available Test Lists. This dropdown has the default All Loaded Tests, which displays all available tests under the project. The other options in the dropdown are Lists of Tests and Tests Not in a List. The List of Tests will display the test lists created using the Test List Editor. It is easier to include the number of tests grouped together and order them. The next option, Tests Not in a List, displays the available tests, which are not part of any Test Lists.

The Available tests list displays all the tests from the test project based on the option chosen in the dropdown.

Selected tests contains the tests that are selected from the available tests list to be placed in order.

The two right and left arrows are used for selecting and unselecting the tests from the Available tests list to the Selected Tests list. We can also select multiple tests by pressing the Ctrl key and selecting the tests.

The up-down arrows on the right of the selected tests list are used for moving up or down the tests and setting the order for the testing in the Selected tests list.

The last option, the Continue after failure checkbox at the bottom of the window, is to override the default behavior of the ordered tests, aborting the execution after the failure of any test. If the option Continue after failure is unchecked, and if any test in the order fails, then all remaining tests will get aborted. In case the tests are not dependent, we can check this option and override the default behavior to allow the application to continue running the remaining tests in order.

Properties of an ordered test

Ordered tests have properties similar to the other test types, in addition to some specific properties. To view the properties, select the ordered test in the Test View or Test List Editor window, right-click and select the Properties option. The Properties dialog box displays the available properties for the ordered test.

Software Testing using Visual Studio Team System 2010

The preceding screenshot shows that most of the properties are the same as the properties of the other test types. We can associate this test with the TFS work items, iterations, and area.

Executing an ordered test

An ordered test can be run like any other test. Open the Test View window or the Test List Editor and select the ordered test from the list, then right-click and choose the Run Selection option from Test View or Run Checked Tests from the Test List Editor. Once the option is selected, we can see the tests running one after the other in the same order in which they are placed in the ordered test. After the execution of the ordered tests, the Test Results window will show the status of the ordered test. If any of the tests in the list fails, then the ordered test status will be Failed. The summary of statuses of all the tests in the ordered test is shown in the following screenshot in the toolbar. The sample ordered test application had four tests in the ordered tests, but two of them failed and one had an error.

Software Testing using Visual Studio Team System 2010

Clicking the Test run failed hyperlink in the status bar shows a detailed view of the test run summary:

Software Testing using Visual Studio Team System 2010

The Test Results window also provides detailed information about the tests run so far. To get these details, choose the test from the Test Results window and then right-click and choose the option, View Test Results Details, which opens the details window and displays the common results information such as Test Name, Result, Duration of the test run, Start Time, End Time, and so on.

Software Testing using Visual Studio Team System 2010

The details window also displays the status of each and every test run within the ordered test. In addition it displays the duration for each test run, name, owner, and type of test in the list. Even though the second test in the list fails, the other tests continue to execute as if the Continue after failure option was checked.

        Read more about this book      

Generic tests

Generic tests are ways to integrate third-party tests into Visual Studio. There could be applications which use third party components or services which have to be tested as part of the whole system testing. In this case, we cannot have our own custom test methods or testing created for the third party component as we may not know the logic and the details of how the component is built. In order to test the third party component, Generic Test types in Visual Studio act as wrappers for executing these third-party tests within the boundary of Visual Studio. Once they are wrapped, we can execute these generic tests like any other test through Visual Studio IDE.

The third-party tests should adhere to the following conditions to be categorized under the generic tests in Visual Studio:

  • We must be able to execute the third-party tests from the command line.
  • The third-party tool must return a Boolean value of either True or False when executed in the command line.
  • It is preferred that the third-party test tool writes an XML file, which contains the list of individual tests. The XML file is the summary results file, which contains the details of all the tests run.

Creating a Generic Test

This is similar to any other test in Visual Studio 2010. Either use the Test menu and choose the option of creating a new test or right-click on the test project in Solution Explorer and add a generic test or open the Test View window and then right-click from there and add a new test. Then provide a name for the generic test. For this example, name it GenericTestforThirdPartyTest. A new window opens to set the values or parameters for the generic test.

(Move the mouse over the image to enlarge.)

You can see from the form that all the values we have to enter are for executing another test application from the command line by passing parameters. For a command line execution—we may have to set the environment variables, execution parameters, set the working directory, copy or deploy some files, and set the output directory and the file. All these details can be set using the generic test.

The following table explains the different options and their uses:

Parameters for Generic Test Description
Specify an existing program This is the name and path of the third-party test application to be executed at the command line; it is the name of the executable application; we can also use the browse button to the right of the text box to find the application and select it.
Command-line arguments to pass to the generic test This is the place to specify the command line parameters required for the third-party test application; these parameters totally depend on the testing tool's expected value.
Additional files to deploy with this generic test In some cases, there might be other files required for this test execution; add those files or remove the selected files in the list using the option to the right of the text box.
Environment variables If the test application used any environment variables for the execution, we are required to set those environment variables for the execution; using this option we can set those environment variables to be used by the test application.
Working directory This is to set the current working directory in the command line before we actually run the test application in the command line.
Redirect standard output and standard error to test result While executing the test application, instead of displaying all the results at the command line, we can redirect those results to the output file, just as we do during the normal command line commands.
Exit test run if run duration (in milliseconds)exceeds This is to limit the wait time for Visual Studio to move on to the next test in the list, or quit; these numbers denotes milliseconds and the default is 60 minutes.
Summary results file This is helpful in case the third-party tests application can write the test results to an XML file; this is the name and path of the XML file in which the output results should be written; if the number of tests in the test application is more in number, then it will be easy to track the result of these individual tests by having the results in the XML file; not only the result but also detailed information of the test result would be written to this file.

The following is an example of a generic test that executes the Test.exe file, which is a third-party test application capable of writing output to the XML file. The command-line parameter for this application is also provided along with the supporting file to be deployed, which is the Readme.txt file. You can see Output.xml, which stores the output details of the test carried out by Test.exe.

Summary results file

When we execute the above generic test, the third-party Test.exe is executed at the command line. The generic test by Visual Studio will get the result back from the third-party Test.exe application, which is a single test. But we do not know how many tests are executed internally within the test, and it is not easy to track the results of all the tests of the third-party application using the generic test. But Visual Studio supports the third-party application with a summary results file, which can be used by the application to write the details of the internal test results.

The third-party applications can make use of the class file, which is generated by using the schema file provided by Visual Studio. The schema file is located at the Visual Studio command-line. If Visual Studio is installed in the default c: drive, then the path would be: C:\Program Files\Microsoft Visual Studio 10.0\Xml\Schemas\SummaryResult.xsd.

The class file can be generated from this schema file using the xsd.exe utility on any .NET supported language. The following is an example of generating the default SummaryResult.cs class file from an XSD file. The output folder should exist before the command is run. c:\temp is the output folder used in the following sample: Xsd SummaryResult.xsd /c /l:cs /out:c:\temp

Software Testing using Visual Studio Team System 2010

The class file is the C# file as we have specifi ed C# as the language in the command-line parameter as /l:cs. The generated output file would be like this:

//-------------------------------
// <auto-generated>
// This code was generated by a tool.
// Runtime Version:4.0.30319.1
//
// Changes to this file may cause incorrect behavior and will be
lost if
// the code is regenerated.
// </auto-generated>
//---------------------------
using System.Xml.Serialization;
//
// This source code was auto-generated by xsd, Version=4.0.30319.1.
//
/// <remarks/>
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd",
"4.0.30319.1")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(AnonymousType=true)]
[System.Xml.Serialization.XmlRootAttribute(Namespace="",
IsNullable=false)]
public partial class SummaryResult {
private string testNameField;
private testResultType testResultField;
private string errorMessageField;
private string detailedResultsFileField;
private SummaryResultInnerTest[] innerTestsField;
/// <remarks/>
public string TestName {
get {
return this.testNameField;
}
set {
this.testNameField = "SampleTest1";
}
}

/// <remarks/>
public testResultType TestResult {
get {
return this.testResultField;
}
set {
this.testResultField = "Passed";
}
}

/// <remarks/>
public string ErrorMessage {
get {
return this.errorMessageField;
}
set {
this.errorMessageField = "Data not in Expected Format";
}
}

/// <remarks/>
public string DetailedResultsFile {
get {
return this.detailedResultsFileField;
}
set {
this.detailedResultsFileField = @"D:\Testing\Trace.txt";
}
}

/// <remarks/>
[System.Xml.Serialization.XmlArrayItemAttribute("InnerTest",
IsNullable=false)]
public SummaryResultInnerTest[] InnerTests {
get {
return this.innerTestsField;
}
set {
this.innerTestsField = value;
}
}
}
/// <remarks/>
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd",
"4.0.30319.1")]
[System.SerializableAttribute()]
public enum testResultType {

/// <remarks/>
Aborted,

/// <remarks/>
Error,

/// <remarks/>
Inconclusive,

/// <remarks/>
Failed,

/// <remarks/>
NotRunnable,

/// <remarks/>
NotExecuted,

/// <remarks/>
Disconnected,

/// <remarks/>
Warning,

/// <remarks/>
InProgress,

/// <remarks/>
Pending,

/// <remarks/>
PassedButRunAborted,

/// <remarks/>
Completed,

/// <remarks/>
Passed,
}

/// <remarks/>
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd",
"4.0.30319.1")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(AnonymousType=true)]
public partial class SummaryResultInnerTest {

private string testNameField;
private testResultType testResultField;
private string errorMessageField;
private string detailedResultsFileField;

/// <remarks/>
public string TestName {
get {
return this.testNameField;
}
set {
this.testNameField = value;
}
}

/// <remarks/>
public testResultType TestResult {
get {
return this.testResultField;
}
set {
this.testResultField = value;
}
}

/// <remarks/>
public string ErrorMessage {
get {
return this.errorMessageField;
}
set {
this.errorMessageField = value;
}
}

/// <remarks/>
public string DetailedResultsFile {
get {
return this.detailedResultsFileField;
}
set {
this.detailedResultsFileField = value;
}
}
}

The third-party tool can make use of the preceding class file to write the test result details, or the test application should take care of writing the test result details into the XML file based on the XML schema used. The resultant output XML file should look like this:

<?xml version="1.0" encoding="utf-8" ?>
<SummaryResult>
<TestName>Third party test Application</TestName>
<TestResult>Failed</TestResult>
<InnerTests>
<InnerTest>
<TestName>Test1</TestName>
<TestResult>Passed</TestResult>
<ErrorMessage>Test is Sucessful</ErrorMessage>
<DetailedResultsFile>
C:\Testing\Test1Results.txt
</DetailedResultsFile>
</InnerTest>
</InnerTests>
</SummaryResult>

In the preceding example, we can see that there are two different tests within a single test. One is to export the data to Excel, which is passed, and the other test is to import the details from Excel, which is failed. The second test which failed writes detailed information about the test result to the text file. Writing into the log file should be taken care of by the third-party test application in the required format.

Now the Test Results window for this generic test would show the result as Passed as shown in the following screenshot:

Software Testing using Visual Studio Team System 2010

Select the result in the Test Results window, right-click and select View Test Results Details, which will show all the summary and detailed results based on the XML output as shown here:

Software Testing using Visual Studio Team System 2010

You can see the Inner Test Results grid where it shows the results of all the inner tests conducted by the third-party test application. If you look at the Test Results Details column, you see the Test1Results.txt file which contains the test result details. Now select the row in the Inner Test Results grid. On selecting the row, the text file Test1Results.txt opens and the details are shown in the Summary File area, which is simply the detailed information written by the application about the failed test:

Software Testing using Visual Studio Team System 2010

The Summary File information is just the content of the text file given in the DetailedResultsFile, in the XML output file.

Properties of a generic test

Besides having some properties in common with the other tests, generic tests also have some specific properties. To view the properties, select the generic test in the Test View or Test List Editor window and right-click to select the Properties option. The Properties dialog box is then displayed with the available properties for the generic test:

Software Testing using Visual Studio Team System 2010

Using the Properties window we can associate a test to a work item, iteration, or a project area defi ned in the TFS. We can also set the input file as arguments, specify deployment items, set the maximum duration for the test run, and many other properties. Most of the properties such as deployment items, description, owner, priority, work item, iteration, and solution are common for all test types, whereas properties such as arguments and target executable are specific to the generic test type.

Summary

This article explained the details and usage of different formats of manual testing using Text and Word. Even though both the tests are similar, the Word format gives better formatting features. The section on the ordered test explained how to order the tests and execute them in the same order irrespective of their type. The generic test explained the ways of executing the third-party tests within Visual Studio and showed the tests results collected within the third-party tests.

Each of these different test types has its own properties. Some of them are common, while some are specific to the individual test. There are some common properties to associate the tests to the work items in the TFS. This is very helpful in mapping the test and tracking a set of related work items.

Books to Consider

comments powered by Disqus