Chapter 13. Testing

In this chapter, we will focus on two aspects of testing for scientific programming. The first aspect is the often difficult topic of what to test in scientific computing. The second aspect covers the question of how to test. We will distinguish between manual and automated testing. Manual testing is what is done by every programmer to quickly check that an implementation is working or not. Automated testing is the refined, automated variant of that idea. We will introduce some tools available for automatic testing in general, with a view on the particular case of scientific computing.

Manual testing

During the development of code, you do a lot of small tests in order to test its functionality. This could be called manual testing. Typically, you would test if a given function does what it is supposed to do, by manually testing the function in an interactive environment. For instance, suppose that you implement the bisection algorithm. It is an algorithm that finds a zero (root) of a scalar non-linear function. To start the algorithm, an interval has to be given with the property that the function takes different signs on the interval boundaries, see Exercise 4, Chapter 7, Functions, for more information.

You will then test an implementation of that algorithm, typically by checking that:

• A solution is found when the function has opposite signs at the interval boundaries
• An exception is raised when the function has the same sign at the interval boundaries

Manual testing, as necessary as it may seem to be, is unsatisfactory. Once you have convinced yourself that the code does...

Automatic testing

The correct way to develop any piece of code is to use automatic testing. The advantages are:

• The automated repetition of a large number of tests after every code refactoring and before any new versions are launched.
• A silent documentation of the use of the code.
• A documentation of the test coverage of your code: Did things work before a change or was a certain aspect never tested?

Note

Changes in the program and in particular in its structure which do not affect its functionality are called code refactoring.

We suggest developing tests in parallel to the code. Good design of tests is an art of its own and there is rarely an investment which guarantees such a good pay-off in development time savings as the investment in good tests.

Now we will go through the implementation of a simple algorithm with the automated testing methods in mind.

Testing the bisection algorithm

Let us examine automated testing for the bisection algorithm. With this algorithm, a zero of a real valued function...

Using unittest package

The standard unittest Python package greatly facilitates automated testing. This package requires that we rewrite our tests to be compatible. The first test would have to be rewritten in a class, as follows:

from bisection import bisect
import unittest

class TestIdentity(unittest.TestCase):
    def test(self):
        result = bisect(lambda x: x, -1.2, 1.,tol=1.e-8)
        expected = 0.
        self.assertAlmostEqual(result, expected)

if __name__=='__main__':
    unittest.main()

Let's examine the differences to the previous implementation. First, the test is now a method and a part of a class. The class must inherit from unittest.TestCase. The test method's name must start with test. Note that we may now use one of the assertion tools of the unittest package, namely assertAlmostEqual. Finally, the tests are run using unittest.main. We recommend to write the tests in a file separate from the code to be tested. That...

Parameterizing tests

One frequently wants to repeat the same test with different data sets. When using the functionalities of unittest this requires us to automatically generate test cases with the corresponding methods injected:

To this end, we first construct a test case with one or several methods that will be used, when we later set up test methods. Let's consider the bisection method again and let's check if the values it returns are really zeros of the given function.

We first build the test case and the method which we will use for the tests as follows:

class Tests(unittest.TestCase):
    def checkifzero(self,fcn_with_zero,interval):
        result = bisect(fcn_with_zero,*interval,tol=1.e-8)
        function_value=fcn_with_zero(result)
        expected=0.
        self.assertAlmostEqual(function_value, expected)

Then we dynamically create test functions as attributes of this class:

test_data=[
           {'name':'identity', 'function':lambda x: x,
   ...

Assertion tools

In this section, we collect the most important tools for raising an AssertionError. We saw the assert command and two tools from unittest, namely assertAlmostEqual. The following table (Table 13.1) summarizes the most important assertion tools and the related modules:

Assertion tool and application example	Module
assert 5==5	–
assertEqual(5.27, 5.27)	unittest.TestCase
assertAlmostEqual(5.24, 5.2,places = 1)	unittest.TestCase
assertTrue(5 > 2)	unittest.TestCase
assertFalse(2 < 5)	unittest.TestCase
assertRaises(ZeroDivisionError,lambda x: 1/x,0.)	unittest.TestCase
assertIn(3,{3,4})	unittest.TestCase
assert_array_equal(A,B)	numpy.testing
assert_array_almost_equal(A, B, decimal=5)	numpy.testing
assert_allclose(A, B, rtol=1.e-3,atol=1.e-5)	numpy.testing

Table 13.1: Assertion tools in Python, unittest and NumPy

Float comparisons

Two floating point numbers should not be compared with the == comparison, because the result of a computation is often slightly off due to rounding errors. There are numerous tools to test equality of floats for testing purposes. First, allclose checks that two arrays are almost equal. It can be used in a test function, as shown:

self.assertTrue(allclose(computed, expected))

Here, self refers to a unittest.Testcase instance. There are also testing tools in the numpy package testing. These are imported by using:

import numpy.testing

Testing that two scalars or two arrays are equal is done using numpy.testing.assert_array_allmost_equal or numpy.testing.assert_allclose. These methods differ in the way they describe the required accuracy, as shown in the preceding table.

QR factorization decomposes a given matrix into a product of an orthogonal matrix Q and an upper triangular matrix R as given in the following example:

import scipy.linalg as sl
A=rand(10,10)
[Q,R]=sl...

Unit and functional tests

Up to now, we have only used functional tests. A functional test checks whether the functionality is correct. For the bisection algorithm, this algorithm actually finds a zero when there is one. In that simple example, it is not really clear what a unit test is. Although, it might seem slightly contrived, it is still possible to make a unit test for the bisection algorithm. It will demonstrate how unit testing often leads to more compartmentalized implementation.

So, in the bisection method, we would like to check, for instance, that at each step the interval is chosen correctly. How to do that? Note that it is absolutely impossible with the current implementation, because the algorithm is hidden inside the function. One possible remedy is to run only one step of the bisection algorithm. Since all the steps are similar, we might argue that we have tested all the possible steps. We also need to be able to inspect the current bounds a and b at the current step of the...

Debugging

Debugging is sometimes necessary while testing, in particular if it is not immediately clear why a given test does not pass. In that case, it is useful to be able to debug a given test in an interactive session. This is however, made difficult by the design of the unittest.TestCase class, which prevents easy instantiation of test case objects. The solution is to create a special instance for debugging purpose only.

Suppose that, in the example of the TestIdentity class above, we want to test the test_functionality method. This would be achieved as follows:

test_case = TestIdentity(methodName='test_functionality')

Now this test can be run individually by:

test_case.debug()

This will run this individual test and it allows for debugging.

Test discovery

If you write a Python package, various tests might be spread out through the package. The discover module finds, imports, and runs these test cases. The basic call from the command line is:

python -m unittest discover

It starts looking for test cases in the current directory and recurses the directory tree downward to find Python objects with the 'test' string contained in its name. The command takes optional arguments. Most important are -s to modify the start directory and -p to define the pattern to recognize the tests:

python -m unittest discover -s '.' -p 'Test*.py'

Measuring execution time

In order to take decisions on code optimization, one often has to compare several code alternatives and decide which code should be preferred based on the execution time. Furthermore, discussing execution time is an issue when comparing different algorithms. In this section, we present a simple and easy way to measure execution time.

Timing with a magic function

The easiest way to measure the execution time of a single statement is to use IPython’s magic function %timeit.

Note

The shell IPython adds additional functionality to standard Python. These extra functions are called magic functions.

As the execution time of a single statement can be extremely short, the statement is placed in a loop and executed several times. By taking the minimum measured time, one makes sure that other tasks running on the computer do not influence the measured result too much. Let's consider four alternative ways to extract nonzero elements from an array as follows:

A=zeros((1000,1000))&...

Summary

No program development without testing! We showed the importance of well organized and documented tests. Some professionals even start development by first specifying tests. A useful tool for automatic testing is the module unittest, which we explained in detail. While testing improves the reliability of a code, profiling is needed to improve the performance. Alternative ways to code may result in large performance differences. We showed how to measure computation time and how to localize bottlenecks in your code.

Exercises

Ex. 1 → Two matrices A, B are called similar, if there exists a matrix S, such that B = S^-1 A S. A and B have the same eigenvalues. Write a test checking that two matrices are similar, by comparing their eigenvalues. Is it a functional or a unit test?

Ex. 2 → Create two vectors of large dimension. Compare the execution time of various ways to compute their dot product:

• SciPy function: dot(v,w)
• Generator and sum: sum((x*y for x,y in zip(v,w)))
• Comprehensive list and sum: sum([x*y for x,y in zip(v,w)])

Ex. 3 → Let u be a vector. The vector v with components

     

 is called a moving average of u. Determine which of the two alternatives to compute v is faster:

v = (u[:-2] + u[1:-1] + u[2:]) / 3

or

v = array([(u[i] + u[i + 1] + u[i + 2]) / 3
  for i in range(len(u)-3)])