Exploring Testing Frameworks

Testing the code is an important part of software development. Although there is no support for testing in the C++ standard, there are a large variety of frameworks for unit testing C++ code. The purpose of this chapter is to get you started with several modern and widely used testing frameworks that enable you to write portable testing code. The frameworks that will be covered in this chapter are Boost.Test, Google Test, and Catch2.

This chapter includes the following recipes:

• Getting started with Boost.Test
• Writing and invoking tests with Boost.Test
• Asserting with Boost.Test
• Using test fixtures with Boost.Test
• Controlling output with Boost.Test
• Getting started with Google Test
• Writing and invoking tests with Google Test
• Asserting with Google Test
• Using test fixtures with Google Test
• Controlling output with Google Test
• Getting started with Catch2
• Writing and invoking tests...

Getting started with Boost.Test

Boost.Test is one of the oldest and most popular C++ testing frameworks. It provides an easy-to-use set of APIs for writing tests and organizing them into test cases and test suites. It has good support for asserting, exception handling, fixtures, and other important features required for a testing framework.

Throughout the next few recipes, we will explore the most important features it has that enable you to write unit tests. In this recipe, we will see how to install the framework and create a simple test project.

Getting ready

The Boost.Test framework has a macro-based API. Although you only need to use the supplied macros for writing tests, a good understanding of macros is recommended if you want to use the framework well.

How to do it...

In order to set up your environment to use Boost.Test, do the following:

1. Download the latest version of the Boost library from http://www.boost.org/.
2. Unzip the content of...

Writing and invoking tests with Boost.Test

The library provides both an automatic and manual way of registering test cases and test suites to be executed by the test runner. Automatic registration is the simplest way because it enables you to construct a test tree just by declaring test units. In this recipe, we will see how to create test suites and test cases using the single-header version of the library, as well as how to run tests.

Getting ready

To exemplify the creation of test suites and test cases, we will use the following class, which represents a three-dimensional point. This implementation contains methods for accessing the properties of a point, comparison operators, a stream output operator, and a method for modifying the position of a point:

class point3d
{
  int x_;
  int y_;
  int z_;
public:
  point3d(int const x = 0, 
          int const y = 0, 
          int const z = 0):x_(x), y_(y), z_(z) {}
  int x() const { return x_; }
  point3d& x(int const...

Asserting with Boost.Test

A test case contains one or more tests. The Boost.Test library provides a series of APIs in the form of macros to write tests. In the previous recipe, you learned a bit about the BOOST_TEST macro, which is the most important and widely used macro in the library. In this recipe, we will discuss how the BOOST_TEST macro can be used in further detail.

Getting ready

You should now be familiar with writing test suites and test cases, a topic we covered in the previous recipe.

How to do it...

The following list shows some of the most commonly used APIs for performing tests:

• BOOST_TEST, in its plain form, is used for most tests:

  int a = 2, b = 4;
  BOOST_TEST(a == b);
  BOOST_TEST(4.201 == 4.200);
  std::string s1{ "sample" };
  std::string s2{ "text" };
  BOOST_TEST(s1 == s2, "not equal");
  

• BOOST_TEST, along with the tolerance() manipulator, is used to indicate the tolerance of floating-point comparisons...

Using fixtures in Boost.Test

The larger a test module is and the more similar the test cases are, the more likely it is to have test cases that require the same setup, cleanup, and maybe the same data. A component that contains these is called a test fixture or test context. Fixtures are important to establish a well-defined environment for running tests so that the results are repeatable. Examples can include copying a specific set of files to some location before executing the tests and deleting them after, or loading data from a particular data source.

Boost.Test provides several ways to define test fixtures for a test case, test suite, or module (globally). In this recipe, we will look at how fixtures work.

Getting ready

The examples in this recipe use the following classes and functions for specifying test unit fixtures:

struct global_fixture
{
   global_fixture()  { BOOST_TEST_MESSAGE("global setup"); }
   ~global_fixture() { BOOST_TEST_MESSAGE("...

Controlling output with Boost.Test

The framework provides us with the ability to customize what is shown in the test log and test report and then format the results. Currently, there are two that are supported: a human-readable format (or HRF) and XML (also with a JUNIT format for the test log). However, it is possible to create and add your own format.

A human-readable format is any form of encoding data that can be naturally read by humans. Text, whether encoded as ASCII or Unicode, is used for this purpose.

The configuration of what is shown in the output can be done both at runtime, through command-line switches, and at compile time, through various APIs. During the execution of the tests, the framework collects all the events in a log. At the end, it produces a report that represents a summary of the execution with different levels of detail. In the case of a failure, the report contains detailed information about the location and the cause, including actual...

Getting started with Google Test

Google Test is one of the most widely used testing frameworks for C++. The Chromium projects and the LLVM compiler are among the projects that are using it for unit testing. Google Test enables developers to write unit tests on multiple platforms using multiple compilers. Google Test is a portable, lightweight framework that has a simple yet comprehensive API for writing tests using asserts; here, tests are grouped into test suites and test suites into test programs.

The framework provides useful features, such as repeating a test a number of times and breaking a test to invoke the debugger at the first failure. Its assertions work regardless of whether exceptions are enabled or not. The next recipe will cover the most important features of the framework. This recipe will show you how to install the framework and set up your first testing project.

Getting ready

The Google Test framework, just like Boost.Test, has a macro-based API. Although...

Writing and invoking tests with Google Test

In the previous recipe, we had a glimpse of what it takes to write simple tests with the Google Test framework. Multiple tests can be grouped into a test suite and one or more test suites grouped into a test program. In this recipe, we will see how to create and run tests.

Getting ready

For the sample code in this recipe, we’ll use the point3d class we discussed in the Writing and invoking tests with Boost.Test recipe.

How to do it...

Use the following macros to create tests:

• TEST(TestSuiteName, TestName) defines a test called TestName as part of a test suite called TestSuiteName:

  TEST(TestConstruction, TestConstructor)
  {
    auto p = point3d{ 1,2,3 };
    ASSERT_EQ(p.x(), 1);
    ASSERT_EQ(p.x(), 2);
    ASSERT_EQ(p.x(), 3);
  }
  TEST(TestConstruction, TestOrigin)
  {
    auto p = point3d::origin();
    ASSERT_EQ(p.x(), 0);
    ASSERT_EQ(p.x(), 0);
    ASSERT_EQ(p.x(), 0);
  }
  

• TEST_F(TestSuiteWithFixture...

Asserting with Google Test

The Google Test framework provides a rich set of both fatal and non-fatal assertion macros, which resemble function calls, to verify the tested code. When these assertions fail, the framework displays the source file, line number, and relevant error message (including custom error messages) to help developers quickly identify the failed code. We have already seen some simple examples of how to use the ASSERT_TRUE macro; in this recipe, we will look at other available macros.

How to do it...

Use the following macros to verify the tested code:

• Use ASSERT_TRUE(condition) or EXPECT_TRUE(condition) to check whether the condition is true, and ASSERT_FALSE(condition) or EXPECT_FALSE(condition) to check whether the condition is false, as shown in the following code:

  EXPECT_TRUE(2 + 2 == 2 * 2);
  EXPECT_FALSE(1 == 2);
  ASSERT_TRUE(2 + 2 == 2 * 2);
  ASSERT_FALSE(1 == 2);
  

• Use ASSERT_XX(val1, val2) or EXPECT_XX(val1, val2) to compare...

Using test fixtures with Google Test

The framework provides support for using fixtures as reusable components for all the tests that are part of a test suite. It also provides support for setting up the global environment in which the tests will run. In this recipe, you will find stepwise instructions on how to define and use test fixtures, as well as to set up the test environment.

Getting ready

You should now be familiar with writing and invoking tests using the Google Test framework, a topic that was covered earlier in this chapter, specifically in the Writing and invoking tests with Google Test recipe.

How to do it...

To create and use a test fixture, do the following:

1. Create a class derived from the testing::Test class:

   class TestFixture : public testing::Test
   {
   };
   

2. Use the constructor to initialize the fixture and the destructor to clean it up:

   protected:
     TestFixture()
     {
       std::cout << "constructing fixture...

Controlling output with Google Test

By default, the output of a Google Test program goes to the standard stream, printed in a human-readable format. The framework provides several options for customizing the output, including printing XML to a disk file in a JUNIT-based format. This recipe will explore the options available to control the output.

Getting ready

For the purpose of this recipe, let’s consider the following test program:

#include <gtest/gtest.h>
TEST(Sample, Test)
{
  auto a = 42;
  ASSERT_EQ(a, 0);
}
int main(int argc, char **argv)
{
  testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
}

Its output is as follows:

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from Sample
[ RUN      ] Sample.Test
f:\chapter11gt_05\main.cpp(6): error: Expected equality of these values:
  a
    Which is: 42
  0
[  FAILED  ] Sample.Test (1 ms)
[----------] 1 test from Sample...

Getting started with Catch2

Catch2 is a multiparadigm testing framework for C++ and Objective-C. The name Catch2 follows on from Catch, the first version of the framework, which stands for C++ Automated Test Cases in Headers. It enables developers to write tests using either the traditional style of test functions grouped in test cases or the behavior-driven development (BDD) style with given-when-then sections. Tests are self-registered and the framework provides several assertion macros; out of these, two are used the most: one fatal (namely, REQUIRE) and one non-fatal (namely, CHECK). They perform expression decomposition of both the left-hand and right-hand side values, which are logged in case of failure. Unlike its first version, Catch2 no longer supports C++03. The current version of Catch2 is v3, which has some significant changes when compared to Catch2 v2, such as the library is no longer a single-header library but works as a regular library (that needs to be compiled)...

Writing and invoking tests with Catch2

The Catch2 framework enables you to write tests using either the traditional style of test cases and test functions or the BDD style with scenarios and given-when-then sections. Tests are defined as separate sections of a test case and can be nested as deep as you want. Whichever style you prefer, tests are defined with only two base macros. This recipe will show what these macros are and how they work.

How to do it...

To write tests using the traditional style, with test cases and test functions, do this:

• Use the TEST_CASE macro to define a test case with a name (as a string), and optionally, a list of its associated tags:

  TEST_CASE("test construction", "[create]")
  {
    // define sections here
  }
  

• Use the SECTION macro to define a test function inside a test case, with the name as a string:

  TEST_CASE("test construction", "[create]")
  {
    SECTION("test constructor...

Asserting with Catch2

Unlike other testing frameworks, Catch2 does not provide a large set of assertion macros. It has two main macros: REQUIRE, which produces a fatal error, stopping the execution of the test case upon failure, and CHECK, which produces a non-fatal error upon failure, continuing the execution of the test case. Several additional macros are defined; in this recipe, we will see how to put them to work.

Getting ready

You should now be familiar with writing test cases and test functions using Catch2, a topic we covered in the previous recipe, Writing and invoking tests with Catch2.

How to do it...

The following list contains the available options for asserting with the Catch2 framework:

• Use CHECK(expr) to check whether expr evaluates to true, continuing the execution in case of failure, and REQUIRE(expr) to make sure that expr evaluates to true, stopping the execution of the test in case of failure:

  int a = 42;
  CHECK(a == 42);
  REQUIRE...

Controlling output with Catch2

As with other testing frameworks discussed in this book, Catch2 reports the results of a test program’s execution in a human-readable format to the stdout standard stream. Additional options are supported, such as reporting using XML format or writing to a file. In this recipe, we will look at the main options available for controlling the output when using Catch2.

Getting ready

To exemplify the way the test program’s execution output could be modified, use the following test cases:

TEST_CASE("case1")
{
  SECTION("function1")
  {
    REQUIRE(true);
  }
}
TEST_CASE("case2")
{
  SECTION("function2")
  {
    REQUIRE(false);
  }
}

The output of running these two test cases is as follows:

----------------------------------------------------------
case2
  function2
----------------------------------------------------------
f:\chapter11ca_04\main.cpp(14)
...............................