Property-Based Testing

Testing is an important aspect of software quality assurance. It exercises the software to expose bugs, ideally early on in the development process where they can be mitigated relatively cheaply. Unfortunately, there are several negative stereotypes associated with testing: it is seen as tedious, time-consuming, and costly. For these reasons, testing is often looked down upon and neglected.

This negative perception is likely due to a particular testing approach: unit testing. Unit testing is rather labor intensive as individual test inputs and expected outputs have to be devised one by one. Typically, a large suite of such unit tests is needed to test the software thoroughly. Some cleverness can go into choosing appropriate inputs, but on the whole, the challenges are limited and the work rather repetitive.

As Haskell programs can also contain bugs, software testing is recommended. However, instead of resorting to unit testing, we have a much more fun and...

Unit testing versus property-based testing

In this section, we’ll review unit testing, probably the most common and well-known form of software testing, and identify several important shortcomings. Then, we’ll introduce property-based testing and explain how it improves upon those shortcomings.

Unit testing

Unit testing is perhaps the most familiar and most obvious way of testing code, especially in the setting of functional programming.

In Haskell, the smallest meaningful unit of code that can be tested is a function. The way to test a function is to supply it with an input and see whether it produces the corresponding expected output. If not, there is either something wrong with the function’s implementation or with our understanding of how it should behave. Either way, we have identified a problem that needs to be investigated further.

Manual testing

With the GHCi interactive shell, such unit testing often happens naturally. When we have written...

Generators

In this section, we’ll teach QuickCheck to generate test inputs for new data types.

Arbitrary

Out of the box, QuickCheck does not support properties of all types. It only supports the common data types in the standard library. Suppose we define a custom data type, say for the two states of a button:

data Button = On | Off deriving (Eq, Show)

We have written a simple function to toggle a button:

toggle :: Button -> Button
toggle On  = Off
toggle Off = On

We want to test this function using the following property:

prop_toggle b = toggle (toggle b) == b

This expresses that toggling the button twice brings it back to its original state. Unfortunately, QuickCheck cannot work with this property:

*Main> quickCheck prop_toggle
<interactive>:3:1: error:
    • No instance for (Arbitrary Button)
        arising from a use of ‘quickCheck’
    • In the expression: quickCheck prop_toggle
      In an equation...

Shrinking

Once QuickCheck has found a counterexample, it’s the tester’s or programmer’s job to figure out why the test fails. The purpose of shrinking is to reduce the size of a counterexample as much as possible to make this more manageable.

Shrinking in action

To see what shrinking does, let’s return to the failing prop_at_zero property. We’ll contrast the same test run. When executed with quickCheck and with verboseCheck, the verbose version of quickCheck gives us a blow-by-blow account of the test run. The first gives a short report:

*Main> quickCheck prop_at_zero
*** Failed! Falsified (after 3 tests and 3 shrinks):
1
[0]

The second also shows the steps that precede the generation of that report:

*Main> verboseCheck prop_at_zero
Failed:
-2
[2,0]
Failed:
2
[2,0]
Passed:
0
[2,0]
Failed:
1
[2,0]
Passed:
0
[2,0]
Passed:
1
[]
Failed:
1
[0]
Passed:
0
[0]
Passed:
1
[]
*** Failed! Falsified (after 3 tests and 3 shrinks):
1
[0]

...

Test properties – a case study

In this section, we’ll explore writing different test properties by employing a small case study.

System under test

Our system under test is a compiler from a small expression language to a corresponding stack language.

We can use the following type of expression:

data Expr
  = Lit Int
  | Add Expr Expr
  | Sub Expr Expr
  | Mul Expr Expr
  deriving Show

It features literals, addition, subtraction, and multiplication.

The stack language features similar functionality, but the parameters of the binary operators are not explicitly given. Instead, they are taken from a stack:

data Instr = Push Int | Plus | Minus | Times
type Prog  = [Instr]
type Stack = [Int]

A program in the stack language is a sequence of instructions that are executed consecutively:

exec :: Prog -> Stack -> Maybe Stack
exec []     s = Just s
exec (i:is) s ...

Summary

In this chapter, we explored testing in Haskell. In particular, we contrasted unit testing with property-based testing in QuickCheck. The latter allows more thorough testing with less effort, but writing test properties requires a bit more abstraction. We have seen that, to support properties of user-defined types, we should instantiate the Arbitrary type class to supply a generator for test inputs. Ideally, we also supply a shrinking strategy, which QuickCheck uses to derive smaller, more manageable counterexamples.

Although still relatively little known outside of the Haskell community, there are clones of the QuickCheck library available for many other programming languages. Also, for Haskell, several variants exist, such as SmallCheck, which replaces random generation with systematic enumeration of small values. Moreover, modern testing frameworks such as hspec combine property-based and unit testing under the same roof.

Questions

Answer the following questions to test your knowledge of this chapter:

1. What are the advantages and disadvantages of property-based testing over unit testing?
2. How does QuickCheck generate test inputs?
3. What is shrinking?

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• The HUnit package documentation:

  https://hackage.haskell.org/package/HUnit

• The QuickCheck package documentation:

  https://hackage.haskell.org/package/QuickCheck

• The hspec package documentation:

  https://hackage.haskell.org/package/hspec

Answers

Here are the answers to this chapter’s questions:

1. Property-based testing has several key advantages over unit testing:
   • Writing a single property requires a little bit more thought than writing a single unit test, but from it, QuickCheck can generate an arbitrary number of unit tests. Hence, property-based testing is more productive.
   • Properties are often a form of documentation of the code under test. Their parametric nature means that they convey insight into many situations, whereas a unit test covers a single situation and thus offers little insight.
   • The random generation of test inputs reveals problems in unexpected corners where human testers would not have looked.

     There are also some disadvantages or costs associated with property-based testing:

   • Writing properties requires thinking at a more abstract level and having more insight into the code. It is debatable whether this is an advantage.
   • There is a setup cost when defining the QuickCheck framework with new data...