Ordered and Generic Tests in Visual Studio 2010

Exclusive offer: get 50% off this eBook here
Software Testing using Visual Studio 2010

Software Testing using Visual Studio 2010 — Save 50%

A step by step guide to understanding the features and concepts of testing applications using Microsoft Visual Studio 2010 with this book and eBook

$26.99    $13.50
by Satheesh Kumar. N Subashni. S | November 2010 | Enterprise Articles Microsoft

This article explains the details of creating and using ordered and generic tests. Visual Studio 2010 provides a feature called Ordered Test to group all or some of these tests and then execute the tests in the same order. The main advantage of creating the ordered test is to execute multiple tests in an order based on the dependencies. For example, Web Performance Tests might depend on the results produced by executing the unit tests. So we need to run these tests in an order so that the Unit Test can be executed before starting the Web Performance Tests.

Let us create sample tests in this article and see the usage of both Generic and Ordered Tests. This article by Satheesh Kumar N and Subashni S, authors of Software Testing using Visual Studio Team System 2010, covers on the following topics:

  • Creating, executing, and learning properties of an Ordered test
  • Creating, executing, and learning properties of an Generic test

 

Software Testing using Visual Studio 2010

Software Testing using Visual Studio 2010

A step by step guide to understand the features and concepts of testing applications using Visual Studio.

  • Master all the new tools and techniques in Visual Studio 2010 and the Team Foundation Server for testing applications
  • Customize reports with Team foundation server.
  • Get to grips with the new Test Manager tool for maintaining Test cases
  • Take full advantage of new Visual Studio features for testing an application's User Interface
  • Packed with real world examples and step by step instructions to get you up and running with application testing

 

        Read more about this book      

Generic Tests are just like any other tests except that they are used for running the existing third party tool or program, which can also be run using command line.

Ordered tests

The following screenshot shows the list of all the tests. You can see that the tests are independent and there is no link between the tests. We have different types of tests like Unit Test, Web Performance Test, and Load Test under the test project. Let's try to create an ordered test and place some of the dependent tests in an order so that the test execution happens in an order without breaking.

Creating an ordered test

There are different ways of creating ordered tests similar to the other tests:

  1. Select the test project from Solution Explorer, right-click and select Add Ordered Test, and then select ordered test from the list of different types of tests. Save the ordered test by choosing the File | Save option.
  2. Select the menu option Test then select New Test..., which opens a dialog with different test types. Select the test type and choose the test project from the Add to Test Project List drop-down and click on OK.

Now the ordered test is created under the test project and the ordered test window is shown to select the existing tests from the test project and set the order.

Software Testing using Visual Studio Team System 2010

The preceding window shows different options for ordering the tests.

The first line is the status bar, which shows the number of tests selected for the ordered test.

The Select test list to view dropdown has the option to choose the display of tests in the available Test Lists. This dropdown has the default All Loaded Tests, which displays all available tests under the project. The other options in the dropdown are Lists of Tests and Tests Not in a List. The List of Tests will display the test lists created using the Test List Editor. It is easier to include the number of tests grouped together and order them. The next option, Tests Not in a List, displays the available tests, which are not part of any Test Lists.

The Available tests list displays all the tests from the test project based on the option chosen in the dropdown.

Selected tests contains the tests that are selected from the available tests list to be placed in order.

The two right and left arrows are used for selecting and unselecting the tests from the Available tests list to the Selected Tests list. We can also select multiple tests by pressing the Ctrl key and selecting the tests.

The up-down arrows on the right of the selected tests list are used for moving up or down the tests and setting the order for the testing in the Selected tests list.

The last option, the Continue after failure checkbox at the bottom of the window, is to override the default behavior of the ordered tests, aborting the execution after the failure of any test. If the option Continue after failure is unchecked, and if any test in the order fails, then all remaining tests will get aborted. In case the tests are not dependent, we can check this option and override the default behavior to allow the application to continue running the remaining tests in order.

Properties of an ordered test

Ordered tests have properties similar to the other test types, in addition to some specific properties. To view the properties, select the ordered test in the Test View or Test List Editor window, right-click and select the Properties option. The Properties dialog box displays the available properties for the ordered test.

Software Testing using Visual Studio Team System 2010

The preceding screenshot shows that most of the properties are the same as the properties of the other test types. We can associate this test with the TFS work items, iterations, and area.

Executing an ordered test

An ordered test can be run like any other test. Open the Test View window or the Test List Editor and select the ordered test from the list, then right-click and choose the Run Selection option from Test View or Run Checked Tests from the Test List Editor. Once the option is selected, we can see the tests running one after the other in the same order in which they are placed in the ordered test. After the execution of the ordered tests, the Test Results window will show the status of the ordered test. If any of the tests in the list fails, then the ordered test status will be Failed. The summary of statuses of all the tests in the ordered test is shown in the following screenshot in the toolbar. The sample ordered test application had four tests in the ordered tests, but two of them failed and one had an error.

Software Testing using Visual Studio Team System 2010

Clicking the Test run failed hyperlink in the status bar shows a detailed view of the test run summary:

Software Testing using Visual Studio Team System 2010

The Test Results window also provides detailed information about the tests run so far. To get these details, choose the test from the Test Results window and then right-click and choose the option, View Test Results Details, which opens the details window and displays the common results information such as Test Name, Result, Duration of the test run, Start Time, End Time, and so on.

Software Testing using Visual Studio Team System 2010

The details window also displays the status of each and every test run within the ordered test. In addition it displays the duration for each test run, name, owner, and type of test in the list. Even though the second test in the list fails, the other tests continue to execute as if the Continue after failure option was checked.

Software Testing using Visual Studio 2010 A step by step guide to understanding the features and concepts of testing applications using Microsoft Visual Studio 2010 with this book and eBook
Published: December 2010
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:
        Read more about this book      

Generic tests

Generic tests are ways to integrate third-party tests into Visual Studio. There could be applications which use third party components or services which have to be tested as part of the whole system testing. In this case, we cannot have our own custom test methods or testing created for the third party component as we may not know the logic and the details of how the component is built. In order to test the third party component, Generic Test types in Visual Studio act as wrappers for executing these third-party tests within the boundary of Visual Studio. Once they are wrapped, we can execute these generic tests like any other test through Visual Studio IDE.

The third-party tests should adhere to the following conditions to be categorized under the generic tests in Visual Studio:

  • We must be able to execute the third-party tests from the command line.
  • The third-party tool must return a Boolean value of either True or False when executed in the command line.
  • It is preferred that the third-party test tool writes an XML file, which contains the list of individual tests. The XML file is the summary results file, which contains the details of all the tests run.

Creating a Generic Test

This is similar to any other test in Visual Studio 2010. Either use the Test menu and choose the option of creating a new test or right-click on the test project in Solution Explorer and add a generic test or open the Test View window and then right-click from there and add a new test. Then provide a name for the generic test. For this example, name it GenericTestforThirdPartyTest. A new window opens to set the values or parameters for the generic test.

(Move the mouse over the image to enlarge.)

You can see from the form that all the values we have to enter are for executing another test application from the command line by passing parameters. For a command line execution—we may have to set the environment variables, execution parameters, set the working directory, copy or deploy some files, and set the output directory and the file. All these details can be set using the generic test.

The following table explains the different options and their uses:

Parameters for Generic Test Description
Specify an existing program This is the name and path of the third-party test application to be executed at the command line; it is the name of the executable application; we can also use the browse button to the right of the text box to find the application and select it.
Command-line arguments to pass to the generic test This is the place to specify the command line parameters required for the third-party test application; these parameters totally depend on the testing tool's expected value.
Additional files to deploy with this generic test In some cases, there might be other files required for this test execution; add those files or remove the selected files in the list using the option to the right of the text box.
Environment variables If the test application used any environment variables for the execution, we are required to set those environment variables for the execution; using this option we can set those environment variables to be used by the test application.
Working directory This is to set the current working directory in the command line before we actually run the test application in the command line.
Redirect standard output and standard error to test result While executing the test application, instead of displaying all the results at the command line, we can redirect those results to the output file, just as we do during the normal command line commands.
Exit test run if run duration (in milliseconds)exceeds This is to limit the wait time for Visual Studio to move on to the next test in the list, or quit; these numbers denotes milliseconds and the default is 60 minutes.
Summary results file This is helpful in case the third-party tests application can write the test results to an XML file; this is the name and path of the XML file in which the output results should be written; if the number of tests in the test application is more in number, then it will be easy to track the result of these individual tests by having the results in the XML file; not only the result but also detailed information of the test result would be written to this file.

The following is an example of a generic test that executes the Test.exe file, which is a third-party test application capable of writing output to the XML file. The command-line parameter for this application is also provided along with the supporting file to be deployed, which is the Readme.txt file. You can see Output.xml, which stores the output details of the test carried out by Test.exe.

Summary results file

When we execute the above generic test, the third-party Test.exe is executed at the command line. The generic test by Visual Studio will get the result back from the third-party Test.exe application, which is a single test. But we do not know how many tests are executed internally within the test, and it is not easy to track the results of all the tests of the third-party application using the generic test. But Visual Studio supports the third-party application with a summary results file, which can be used by the application to write the details of the internal test results.

The third-party applications can make use of the class file, which is generated by using the schema file provided by Visual Studio. The schema file is located at the Visual Studio command-line. If Visual Studio is installed in the default c: drive, then the path would be: C:\Program Files\Microsoft Visual Studio 10.0\Xml\Schemas\SummaryResult.xsd.

The class file can be generated from this schema file using the xsd.exe utility on any .NET supported language. The following is an example of generating the default SummaryResult.cs class file from an XSD file. The output folder should exist before the command is run. c:\temp is the output folder used in the following sample: Xsd SummaryResult.xsd /c /l:cs /out:c:\temp

Software Testing using Visual Studio Team System 2010

The class file is the C# file as we have specifi ed C# as the language in the command-line parameter as /l:cs. The generated output file would be like this:

//-------------------------------
// <auto-generated>
// This code was generated by a tool.
// Runtime Version:4.0.30319.1
//
// Changes to this file may cause incorrect behavior and will be
lost if
// the code is regenerated.
// </auto-generated>
//---------------------------
using System.Xml.Serialization;
//
// This source code was auto-generated by xsd, Version=4.0.30319.1.
//
/// <remarks/>
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd",
"4.0.30319.1")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(AnonymousType=true)]
[System.Xml.Serialization.XmlRootAttribute(Namespace="",
IsNullable=false)]
public partial class SummaryResult {
private string testNameField;
private testResultType testResultField;
private string errorMessageField;
private string detailedResultsFileField;
private SummaryResultInnerTest[] innerTestsField;
/// <remarks/>
public string TestName {
get {
return this.testNameField;
}
set {
this.testNameField = "SampleTest1";
}
}

/// <remarks/>
public testResultType TestResult {
get {
return this.testResultField;
}
set {
this.testResultField = "Passed";
}
}

/// <remarks/>
public string ErrorMessage {
get {
return this.errorMessageField;
}
set {
this.errorMessageField = "Data not in Expected Format";
}
}

/// <remarks/>
public string DetailedResultsFile {
get {
return this.detailedResultsFileField;
}
set {
this.detailedResultsFileField = @"D:\Testing\Trace.txt";
}
}

/// <remarks/>
[System.Xml.Serialization.XmlArrayItemAttribute("InnerTest",
IsNullable=false)]
public SummaryResultInnerTest[] InnerTests {
get {
return this.innerTestsField;
}
set {
this.innerTestsField = value;
}
}
}
/// <remarks/>
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd",
"4.0.30319.1")]
[System.SerializableAttribute()]
public enum testResultType {

/// <remarks/>
Aborted,

/// <remarks/>
Error,

/// <remarks/>
Inconclusive,

/// <remarks/>
Failed,

/// <remarks/>
NotRunnable,

/// <remarks/>
NotExecuted,

/// <remarks/>
Disconnected,

/// <remarks/>
Warning,

/// <remarks/>
InProgress,

/// <remarks/>
Pending,

/// <remarks/>
PassedButRunAborted,

/// <remarks/>
Completed,

/// <remarks/>
Passed,
}

/// <remarks/>
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd",
"4.0.30319.1")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(AnonymousType=true)]
public partial class SummaryResultInnerTest {

private string testNameField;
private testResultType testResultField;
private string errorMessageField;
private string detailedResultsFileField;

/// <remarks/>
public string TestName {
get {
return this.testNameField;
}
set {
this.testNameField = value;
}
}

/// <remarks/>
public testResultType TestResult {
get {
return this.testResultField;
}
set {
this.testResultField = value;
}
}

/// <remarks/>
public string ErrorMessage {
get {
return this.errorMessageField;
}
set {
this.errorMessageField = value;
}
}

/// <remarks/>
public string DetailedResultsFile {
get {
return this.detailedResultsFileField;
}
set {
this.detailedResultsFileField = value;
}
}
}

The third-party tool can make use of the preceding class file to write the test result details, or the test application should take care of writing the test result details into the XML file based on the XML schema used. The resultant output XML file should look like this:

<?xml version="1.0" encoding="utf-8" ?>
<SummaryResult>
<TestName>Third party test Application</TestName>
<TestResult>Failed</TestResult>
<InnerTests>
<InnerTest>
<TestName>Test1</TestName>
<TestResult>Passed</TestResult>
<ErrorMessage>Test is Sucessful</ErrorMessage>
<DetailedResultsFile>
C:\Testing\Test1Results.txt
</DetailedResultsFile>
</InnerTest>
</InnerTests>
</SummaryResult>

In the preceding example, we can see that there are two different tests within a single test. One is to export the data to Excel, which is passed, and the other test is to import the details from Excel, which is failed. The second test which failed writes detailed information about the test result to the text file. Writing into the log file should be taken care of by the third-party test application in the required format.

Now the Test Results window for this generic test would show the result as Passed as shown in the following screenshot:

Software Testing using Visual Studio Team System 2010

Select the result in the Test Results window, right-click and select View Test Results Details, which will show all the summary and detailed results based on the XML output as shown here:

Software Testing using Visual Studio Team System 2010

You can see the Inner Test Results grid where it shows the results of all the inner tests conducted by the third-party test application. If you look at the Test Results Details column, you see the Test1Results.txt file which contains the test result details. Now select the row in the Inner Test Results grid. On selecting the row, the text file Test1Results.txt opens and the details are shown in the Summary File area, which is simply the detailed information written by the application about the failed test:

Software Testing using Visual Studio Team System 2010

The Summary File information is just the content of the text file given in the DetailedResultsFile, in the XML output file.

Properties of a generic test

Besides having some properties in common with the other tests, generic tests also have some specific properties. To view the properties, select the generic test in the Test View or Test List Editor window and right-click to select the Properties option. The Properties dialog box is then displayed with the available properties for the generic test:

Software Testing using Visual Studio Team System 2010

Using the Properties window we can associate a test to a work item, iteration, or a project area defi ned in the TFS. We can also set the input file as arguments, specify deployment items, set the maximum duration for the test run, and many other properties. Most of the properties such as deployment items, description, owner, priority, work item, iteration, and solution are common for all test types, whereas properties such as arguments and target executable are specific to the generic test type.

Summary

This article explained the details and usage of different formats of manual testing using Text and Word. Even though both the tests are similar, the Word format gives better formatting features. The section on the ordered test explained how to order the tests and execute them in the same order irrespective of their type. The generic test explained the ways of executing the third-party tests within Visual Studio and showed the tests results collected within the third-party tests.

Each of these different test types has its own properties. Some of them are common, while some are specific to the individual test. There are some common properties to associate the tests to the work items in the TFS. This is very helpful in mapping the test and tracking a set of related work items.

Software Testing using Visual Studio 2010 A step by step guide to understanding the features and concepts of testing applications using Microsoft Visual Studio 2010 with this book and eBook
Published: December 2010
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

About the Author :


Satheesh Kumar. N

Satheesh Kumar N holds a Bachelor’s degree in Computer Science engineering and has around 17 years of experience in managing the software development life cycle, developing live projects, and program management. He started his career by developing software applications using Borland software products. He worked for multiple organizations in India, the UAE, and the US. His main domain expertise is in retail and he is currently working in Bangalore as a Program Delivery Manager for the top retailer in UK. He is currently handling five agile scrum teams for delivering the website features. His experience also includes implementation and customization of Microsoft Dynamics for an automobile sales company in UAE. He works with the latest Microsoft technologies and has published many articles on LINQ and other features of .NET. He is a certified PMP (Project Management Professional).

He has also authored Software Testing using Visual Studio Team System 2008 and Software Testing using Visual Studio 2010 for Packt Publishing.

Subashni. S

Subashni S holds a Bachelor’s Degree in Computer Science engineering and has around 15 years of experience in software development and testing life cycle, project, and program management. She is a certified PMP (Project Management Professional), CSTM (Certified Software Test Manager), and ITIL V3 Foundation certified. She started her career as a DBA in Oracle 8i technology, and later developed many software applications using Borland software products for a multinational company based in Chennai, and then moved to Bangalore. She is presently working for a multinational company, in the area of Project Management for developing and testing projects. She is also currently working for one of the top multinational companies headquartered at Austin, Texas.

She has also authored Software Testing using Visual Studio Team System 2008 and Software Testing using Visual Studio 2010 for Packt Publishing.

Books From Packt


Microsoft Visio 2010 Business Process Diagramming and Validation
Microsoft Visio 2010 Business Process Diagramming and Validation

Microsoft Windows Workflow Foundation 4.0 Cookbook
Microsoft Windows Workflow Foundation 4.0 Cookbook

Refactoring with Microsoft Visual Studio 2010
Refactoring with Microsoft Visual Studio 2010

Microsoft Dynamics GP 2010 Implementation
Microsoft Dynamics GP 2010 Implementation

Microsoft Dynamics NAV Administration
Microsoft Dynamics NAV Administration

Applied Architecture Patterns on the Microsoft Platform
Applied Architecture Patterns on the Microsoft Platform

Microsoft SQL Server 2008 High Availability
Microsoft SQL Server 2008 High Availability

NHibernate 3.0 Cookbook
NHibernate 3.0 Cookbook


Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software