Testing

In this chapter, we'll focus on two aspects of testing for scientific programming. The first aspect is the often difficult topic of what to test in scientific computing. The second aspect covers the question of how to test. We will distinguish between manual and automated testing. Manual testing is what is done by every programmer to quickly check that a program is doing what it should or should not. Automated testing is the refined, automated variant of that idea. We will introduce some tools available for automatic testing in general, with a view of the particular case of scientific computing.

15.1 Manual testing

During code development, you do a lot of small tests in order to test its functionality. This could be called manual testing. Typically, you would test if a given function does what it is supposed to do, by manually testing the function in an interactive environment. For instance, suppose that you implement the bisection algorithm. It is an algorithm that finds a zero (root) of a scalar non-linear function. To start the algorithm, an interval has to be given with the property that the function takes different signs on the interval boundaries (see Exercise 4 in Section 7.10: Exercises, for more information).

You will then test an implementation of that algorithm, typically by checking that:

• A solution is found when the function has opposite signs at the interval boundaries.
• An exception is raised when the function has the same sign at the interval boundaries.

Manual testing, as necessary as it may seem to be, is unsatisfactory. Once you have convinced yourself...

15.2 Automatic testing

The correct way to develop any piece of code is to use automatic testing. The advantages are:

• The automated repetition of a large number of tests after every code refactoring and before any new versions are launched.
• Silent documentation of the use of the code.
• Documentation of the test coverage of your code: Did things work before a change or was a certain aspect never tested?

Changes in the program and in particular in its structure that do not affect its functionality are called code refactoring.

We suggest developing tests in parallel to coding. Good design of tests is an art of its own and there is rarely an investment that guarantees such a good pay-off in development time savings as the investment in good tests.

Now we will go through the implementation of a simple algorithm with the automated testing methods in mind.

15.2.1 Testing the bisection algorithm

Let's examine automated testing for the bisection algorithm. With this algorithm, a zero of a real-valued function is found. It is described in Exercise 4 in Section 7.10: Exercises. An implementation of the algorithm can have the following form:

def bisect(f, a, b, tol=1.e-8):
    """
    Implementation of the bisection algorithm 
    f real valued function
    a,b interval boundaries (float) with the property 
    f(a) * f(b) <= 0
    tol tolerance (float)
    """
    if f(a) * f(b)> 0:
        raise ValueError("Incorrect initial interval [a, b]") 
    for i in range(100):
        c = (a + b) / 2.
        if f(a) * f(c) <= 0:
            b = c
        else:
            a = c
        if abs(a - b) < tol:
            return (a + b) / 2
    raise Exception('No root found within the given tolerance {tol}')

We assume this to be stored in a file named bisection...

15.2.2 Using the unittest module

The Python module unittest greatly facilitates automated testing. This module requires that we rewrite our previous tests to be compatible. 

The first test would have to be rewritten in a class, as follows:

from bisection import bisect
import unittest

class TestIdentity(unittest.TestCase):
    def test(self):
        result = bisect(lambda x: x, -1.2, 1.,tol=1.e-8)
        expected = 0.
        self.assertAlmostEqual(result, expected)

if __name__=='__main__':
    unittest.main()

Let's examine the differences from the previous implementation. First, the test is now a method and a part of a class. The class must inherit from unittest.TestCase. The test method's name must start with test. Note that we may now use one of the assertion tools of the unittest package, namely assertAlmostEqual. Finally, the tests are run using unittest.main. We recommend writing the tests in a file...

15.2.3 Test setUp and tearDown methods

The class unittest.TestCase provides two special methods, setUp and tearDown, which run before and after every call to a test method. This is needed when testing generators, which are exhausted after every test. We demonstrate this by testing a program that checks the line in a file in which a given string occurs for the first time:

class StringNotFoundException(Exception):
    pass

def find_string(file, string):
    for i,lines in enumerate(file.readlines()):
        if string in lines:
            return i
    raise StringNotFoundException(
          f'String {string} not found in File {file.name}.')

We assume that this code is saved in a file named find_in_file.py.

A test has to prepare a file and open it and remove it after the test as given in the following example:

import unittest
import os # used for, for example, deleting files

from find_in_file import find_string, StringNotFoundException...

Setting up testdata when a test case is created

The methods setUp and tearDown are executed before and after any test method of a test case. This is necessary when the test methods change the data. They guarantee that the test data is restored before the next test is executed.

However, there is also often a situation where your tests do not change the test data and you want to save time by only once setting up the data. This is done by the class method setUpClass.

The following code block schematically illustrates how the method setUpClass is used. You might also want to check Section 8.4: Class attributes and class methods again.

 

import unittest

class TestExample(unittest.Testcase):
     @classmethod
     def setUpClass(cls):
         cls.A=....
     def Test1(self):
         A=self.A
         # assert something
         ....
     def Test2(self):
         A=self.A
         # assert something else

15.2.4 Parameterizing tests

We frequently want to repeat the same test with different datasets. When using the functionalities of unittest, this requires us to automatically generate test cases with the corresponding methods injected.

To this end, we first construct a test case with one or several methods that will be used, when we later set up test methods. We'll consider the bisection method again and let's check if the values it returns are really zeros of the given function.

We first build the test case and the method that we will use for the tests as follows:

class Tests(unittest.TestCase):
    def checkifzero(self,fcn_with_zero,interval):
        result = bisect(fcn_with_zero,*interval,tol=1.e-8)
        function_value=fcn_with_zero(result)
        expected=0.
        self.assertAlmostEqual(function_value, expected)

Then we dynamically create test functions as attributes of this class:

test_data=[
           {'name':'identity', 'function...

15.2.5 Assertion tools

In this section, we collect the most important tools for raising an AssertionError. We saw the command assert and three tools from unittest, namely assertAlmostEqual, assertEqual, and assertRaises. The following table (Table 15.1) summarizes the most important assertion tools and the related modules:

Assertion tool and application example	Module
assert 5==5	–
assertEqual(5.27, 5.27)	unittest.TestCase
assertAlmostEqual(5.24, 5.2,places = 1)	unittest.TestCase
assertTrue(5 > 2)	unittest.TestCase
assertFalse(2 < 5)	unittest.TestCase
assertRaises(ZeroDivisionError,lambda x: 1/x,0.)	unittest.TestCase
assertIn(3,{3,4})	unittest.TestCase
assert_array_equal(A,B)	numpy.testing
assert_array_almost_equal(A, B, decimal=5)	numpy.testing
assert_allclose(A, B, rtol=1.e-3,atol=1.e-5)	numpy.testing

Table 15.1: Assertion tools in Python, unittest, and...

15.2.6 Float comparisons

Two floating-point numbers should not be compared with the == comparison, because the result of a computation is often slightly off due to rounding errors. There are numerous tools to test the equality of floats for testing purposes.

First, allclose checks that two arrays are almost equal. It can be used in a test function, as shown:

self.assertTrue(allclose(computed, expected))

Here, self refers to a unittest.Testcase instance. There are also testing tools in the numpy package testing. These are imported by using:

import numpy.testing

Testing that two scalars or two arrays are equal is done using numpy.testing.assert_array_allmost_equal or numpy.testing.assert_allclose. These methods differ in the way they describe the required accuracy, as shown in the preceding table, Table 15.1.

 factorization decomposes a given matrix into a product of an orthogonal matrix  and an upper triangular...

15.2.7 Unit and functional tests

Up to now, we have only used functional tests. A functional test checks whether the functionality is correct. For the bisection algorithm, this algorithm actually finds a zero when there is one. In that simple example, it is not really clear what a unit test is. Although it might seem slightly contrived, it is still possible to make a unit test for the bisection algorithm. It will demonstrate how unit testing often leads to more compartmentalized implementation.

So, in the bisection method, we would like to check, for instance, that at each step the interval is chosen correctly. How to do that? Note that it is absolutely impossible with the current implementation because the algorithm is hidden inside the function. One possible remedy is to run only one step of the bisection algorithm. Since all the steps are similar, we might argue that we have tested all the possible steps. We also need to be able to inspect the current bounds ...

15.2.8 Debugging

Debugging is sometimes necessary while testing, in particular, if it is not immediately clear why a given test does not pass. In that case, it is useful to be able to debug a given test in an interactive session. This is, however, made difficult by the design of the class unittest.TestCase, which prevents easy instantiation of test case objects. The solution is to create a special instance for debugging purposes only.

Suppose that, in the previous example of the class TestIdentity, we want to test the method test_functionality. This would be achieved as follows:

test_case = TestIdentity(methodName='test_functionality')

Now this test can be run individually with:

test_case.debug()

This will run this individual test and it allows for debugging.

15.2.9 Test discovery

If you write a Python package, various tests might be spread out through the package. The module discover finds, imports, and runs these test cases. The basic call from the command line is:

python -m unittest discover

It starts looking for test cases in the current directory and recurses the directory tree downward to find Python objects with the 'test' string contained in its name. The command takes optional arguments. Most important are -s to modify the start directory and -p to define the pattern to recognize the tests:

python -m unittest discover -s '.' -p 'Test*.py'

15.3 Measuring execution time

In order to take decisions on code optimization, you often have to compare several code alternatives and decide which code should be preferred based on the execution time. Furthermore, discussing execution time is an issue when comparing different algorithms. In this section, we present a simple and easy way to measure execution time.

15.3.1 Timing with a magic function

The easiest way to measure the execution time of a single statement is to use IPython’s magic function %timeit.

The shell IPython adds additional functionality to standard Python. These extra functions are called magic functions.

As the execution time of a single statement can be extremely short, the statement is placed in a loop and executed several times. By taking the minimum measured time, you make sure that other tasks running on the computer do not influence the measured result too much.

Let's consider four alternative ways to extract nonzero elements from an array as follows:

A=zeros((1000,1000))
A[53,67]=10

def find_elements_1(A):
    b = []
    n, m = A.shape
    for i in range(n):
        for j in range(m):
            if abs(A[i, j]) > 1.e-10:
                b.append(A[i, j])
    return b

def find_elements_2(A):
    return [a for a in A.reshape((-1, )) if abs(a) > 1.e-10]

def find_elements_3...

15.3.2 Timing with the Python module timeit

Python provides the module timeit, which can be used to measure execution time. It requires that, first, a time object is constructed. It is constructed from two strings: a string with setup commands and a string with the commands to be executed.

We take the same four alternatives as in the preceding example. The array and function definitions are written now in a string called setup_statements and four timing objects are constructed as follows:

import timeit
setup_statements="""
from scipy import zeros
from numpy import where
A=zeros((1000,1000))
A[57,63]=10.

def find_elements_1(A):
    b = []
    n, m = A.shape
    for i in range(n):
        for j in range(m):
            if abs(A[i, j]) > 1.e-10:
               b.append(A[i, j])
    return b

def find_elements_2(A):
    return [a for a in A.reshape((-1,)) if abs(a) > 1.e-10]

def find_elements_3(A):
    return [a for a in A.flatten() if...

15.3.3 Timing with a context manager

Finally, we present the third method. It serves to show another application of a context manager. We first construct a context manager object for measuring the elapsed time as shown:

import time
class Timer:
    def __enter__(self):
        self.start = time.time()
        # return self
    def __exit__(self, ty, val, tb):
        end = time.time()
        self.elapsed=end-self.start
        print(f'Time elapsed {self.elapsed} seconds') 
        return False

Recall that the __enter__ and __exit__ methods make this class a context manager. The __exit__ method's parameters ty, val, and tb are in the normal case None. If an exception is raised during execution, they take the exception type, its value, and traceback information. The return value False indicates that the exception has not been caught so far.

We'll now show the use of the ...

15.4 Summary

No program development without testing! We showed the importance of well-organized and documented tests. Some professionals even start development by first specifying tests. A useful tool for automatic testing is the module unittest, which we explained in detail. While testing improves the reliability of code, profiling is needed to improve the performance. Alternative ways to code may result in large performance differences. We showed how to measure computation time and how to localize bottlenecks in your code.

15.5 Exercises

Ex. 1: Two matrices  are called similar if there exists a matrix , such that . The matrices  and  have the same eigenvalues. Write a test checking that two matrices are similar, by comparing their eigenvalues. Is it a functional or a unit test?

Ex. 2: Create two vectors of large dimensions. Compare the execution time of various ways to compute their dot product:

• SciPy function: v @ w
  
• Generator and sum: sum((x*y for x,y in zip(v,w)))
• Comprehensive list and sum: sum([x*y for x,y in zip(v,w)])

Ex. 3: Let be a vector. The vector  with components

 is called a moving average of . Determine which of the two alternatives to compute  is faster:

v = (u[:-2] + u[1:-1] + u[2:]) / 3

or

v = array([(u[i] + u[i + 1] + u[i + 2]) / 3
  for i in range(len(u)-3)])