Testing in Python using doctest

Exclusive offer: get 50% off this eBook here
Python Testing Cookbook

Python Testing Cookbook — Save 50%

Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools

$26.99    $13.50
by Greg L. Turnquist | May 2011 | Open Source

Testing has always been a part of software development. Testing includes many different styles including unit testing, integration testing, acceptance testing, smoke testing, load testing, and countless others.

In this article by Greg Lee Turnquist, author of Python Testing Cookbook, we will cover:

  • Coding a test harness for doctest
  • Filtering out test noise
  • Testing the edges
  • Testing corner cases by iteration
  • Getting nosy with doctest

 

Python Testing Cookbook

Python Testing Cookbook

Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools

        Read more about this book      

(For more resources on Python, see here.)

The reader can benefit from the previous article on Python: Using doctest for Documentation.

 

Coding a test harness for doctest

The doctest module supports creating objects, invoking methods, and checking results. With this recipe, we will explore this in more detail.

An important aspect of doctest is that it finds individual instances of docstrings, and runs them in a local context. Variables declared in one docstring cannot be used in another docstring.

How to do it...

  1. Create a new file called recipe19.py to contain the code from this recipe.
  2. Write a simple shopping cart application.

    class ShoppingCart(object):
    def __init__(self):
    self.items = []
    def add(self, item, price):
    self.items.append(Item(item, price))
    return self
    def item(self, index):
    return self.items[index-1].item
    def price(self, index):
    return self.items[index-1].price
    def total(self, sales_tax):
    sum_price = sum([item.price for item in self.items])
    return sum_price*(1.0 + sales_tax/100.0)
    def __len__(self):
    return len(self.items)

    class Item(object):
    def __init__(self, item, price):
    self.item = item
    self.price = price

  3. Insert a docstring at the top of the module, before the ShoppingCart class declaration.
    """
    This is documentation for the this entire recipe.
    With it, we can demonstrate usage of the code.

    >>> cart = ShoppingCart().add("tuna sandwich", 15.0)
    >>> len(cart)
    1
    >>> cart.item(1)
    'tuna sandwich'
    >>> cart.price(1)
    15.0
    >>> print round(cart.total(9.25), 2)
    16.39
    """
    class ShoppingCart(object):
    ...
  4. Run the recipe using -m doctest and -v for verbosity.

    Python Testing Cookbook

  5. Copy all the code we just wrote from recipe19.py into a new file called recipe19b.py.
  6. Inside recipe19b.py add another docstring to item, which attempts to re-use the cart variable defined at the top of the module.

    def item(self, index):
    """
    >>> cart.item(1)
    'tuna sandwich'
    """
    return self.items[index-1].item

  7. Run this variant of the recipe. Why does it fail? Wasn't cart declared in the earlier docstring?

    Python Testing Cookbook

How it works...

The doctest module looks for every docstring. For each docstring it finds, it creates a shallow copy of the module's global variables and then runs the code and checks results. Apart from that, every variable created is locally scoped and then cleaned up when the test is complete. This means that our second docstring that was added later cannot see the cart that was created in our first docstring. That is why the second run failed.

There is no equivalent to a setUp method as we used with some of the unittest recipes. If there is no setUp option with doctest, then what value is this recipe? It highlights a key limitation of doctest that all developers must understand before using it.

There's more...

The doctest module provides an incredibly convenient way to add testability to our documentation. But this is not a substitute for a full-fledged testing framework, like unittest. As noted earlier, there is no equivalent to a setUp. There is also no syntax checking of the Python code embedded in the docstrings.

Mixing the right level of doctests with unittest (or other testing framework we pick) is a matter of judgment.

 

Filtering out test noise

Various options help doctest ignore noise, such as whitespace, in test cases. This can be useful, because it allows us to structure the expected outcome in a better way, to ease reading for the users.

We can also flag some tests that can be skipped. This can be used where we want to document known issues, but haven't yet patched the system.

Both of these situations can easily be construed as noise, when we are trying to run comprehensive testing, but are focused on other parts of the system. In this recipe, we will dig in to ease the strict checking done by doctest. We will also look at how to ignore entire tests, whether it's on a temporary or permanent basis.

How to do it...

With the following steps, we will experiment with filtering out test results and easing certain restrictions of doctest.

  1. Create a new file called recipe20.py to contain the code from this recipe.
  2. Create a recursive function that converts base10 numbers into other bases.

    def convert_to_basen(value, base):
    import math
    def _convert(remaining_value, base, exp):
    def stringify(value):
    if value > 9:
    return chr(value + ord('a')-10)
    else:
    return str(value)

    if remaining_value >= 0 and exp >= 0:
    factor = int(math.pow(base, exp))
    if factor <= remaining_value:
    multiple = remaining_value / factor
    return stringify(multiple) + \
    _convert(remaining_value-multiple*factor, \
    base, exp-1)
    else:
    return "0" + \
    _convert(remaining_value, base, exp-1)
    else:
    return ""
    return "%s/%s" % (_convert(value, base, \
    int(math.log(value, base))), base)

  3. Add a docstring that includes a test to exercise a range of values as well as documenting a future feature that is not yet implemented.

    def convert_to_basen(value, base):
    """Convert a base10 number to basen.
    >>> [convert_to_basen(i, 16) for i in range(1,16)] #doctest:
    +NORMALIZE_WHITESPACE
    ['1/16', '2/16', '3/16', '4/16', '5/16', '6/16', '7/16',
    '8/16',
    '9/16', 'a/16', 'b/16', 'c/16', 'd/16', 'e/16', 'f/16']
    FUTURE: Binary may support 2's complement in the future, but
    not now.
    >>> convert_to_basen(-10, 2) #doctest: +SKIP
    '0110/2'
    """
    import math

  4. Add a test runner.

    if __name__ == "__main__":
    import doctest
    doctest.testmod()

  5. Run the test case in verbose mode.

    Python Testing Cookbook

  6. Copy the code from recipe20.py into a new file called recipe20b.py.
  7. Edit recipe20b.py by updating the docstring to include another test exposing that our function doesn't convert 0.

    def convert_to_basen(value, base):
    """Convert a base10 number to basen.
    >>> [convert_to_basen(i, 16) for i in range(1,16)] #doctest:
    +NORMALIZE_WHITESPACE
    ['1/16', '2/16', '3/16', '4/16', '5/16', '6/16', '7/16',
    '8/16',
    '9/16', 'a/16', 'b/16', 'c/16', 'd/16', 'e/16', 'f/16']
    FUTURE: Binary may support 2's complement in the future, but
    not now.
    >>> convert_to_basen(-10, 2) #doctest: +SKIP
    '0110/2'
    BUG: Discovered that this algorithm doesn't handle 0. Need to
    patch it.
    TODO: Renable this when patched.
    >>> convert_to_basen(0, 2)
    '0/2'
    """
    import math

  8. Run the test case. Notice what is different about this version of the recipe; and why does it fail?

    Python Testing Cookbook

  9. Copy the code from recipe20b.py into a new file called recipe20c.py.
  10. Edit recipe20c.py and update the docstring indicating that we will skip the test for now.

    def convert_to_basen(value, base):
    """Convert a base10 number to basen.
    >>> [convert_to_basen(i, 16) for i in range(1,16)] #doctest:
    +NORMALIZE_WHITESPACE
    ['1/16', '2/16', '3/16', '4/16', '5/16', '6/16', '7/16',
    '8/16',
    '9/16', 'a/16', 'b/16', 'c/16', 'd/16', 'e/16', 'f/16']
    FUTURE: Binary may support 2's complement in the future, but
    not now.
    >>> convert_to_basen(-10, 2) #doctest: +SKIP
    '0110/2'
    BUG: Discovered that this algorithm doesn't handle 0. Need to
    patch it.
    TODO: Renable this when patched.
    >>> convert_to_basen(0, 2) #doctest: +SKIP
    '0/2'
    """
    import math

  11. Run the test case.

    Python Testing Cookbook

How it works...

In this recipe, we revisit the function for converting from base-10 to any base numbers. The first test shows it being run over a range. Normally, Python would fit this array of results on one line. To make it more readable, we spread the output across two lines. We also put some arbitrary spaces between the values to make the columns line up better.

This is something that doctest definitely would not support, due to its strict pattern matching nature. By using #doctest: +NORMALIZE_WHITESPACE, we are able to ask doctest to ease this restriction. There are still constraints. For example, the first value in the expected array cannot have any whitespace in front of it. But wrapping the array to the next line no longer breaks the test.

We also have a test case that is really meant as documentation only. It indicates a future requirement that shows how our function would handle negative binary values. By adding #doctest: +SKIP, we are able to command doctest to skip this particular instance.

Finally, we see the scenario where we discover that our code doesn't handle 0. As the algorithm gets the highest exponent by taking a logarithm, there is a math problem. We capture this edge case with a test. We then confirm that the code fails in classic test driven design (TDD) fashion. The final step would be to fix the code to handle this edge case. But we decide, in a somewhat contrived fashion, that we don't have enough time in the current sprint to fix the code. To avoid breaking our continuous integration (CI) server, we mark the test with a TO-DO statement and add #doctest: +SKIP.

There's more...

Both the situations that we have marked up with #doctest: +SKIP, are cases where eventually we will want to remove the SKIP tag and have them run. There may be other situations where we will never remove SKIP. Demonstrations of code that have big fluctuations may not be readily testable without making them unreadable. For example, functions that return dictionaries are harder to test, because the order of results varies. We can bend it to pass a test, but we may lose the value of documentation to make it presentable to the reader.

 

Python Testing Cookbook Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools
Published: May 2011
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

 

        Read more about this book      

(For more resources on Python, see here.)

 

Testing the edges

Tests need to exercise the boundaries of our code up to and beyond the range limits. In this recipe, we will dig into defining and testing edges with doctest.

How to do it...

With the following steps, we will see how to write code that tests the edges of our software.

  1. Create a new file named recipe22.py and use it to store all of our code for this recipe.
  2. Create a function that converts base 10 numbers to anything between base 2 and base 36.

    def convert_to_basen(value, base):
    if base < 2 or base > 36:
    raise Exception("Only support bases 2-36")
    import math
    def _convert(remaining_value, base, exp):
    def stringify(value):
    if value > 9:
    return chr(value + ord('a')-10)
    else:
    return str(value)
    if remaining_value >= 0 and exp >= 0:
    factor = int(math.pow(base, exp))
    if factor <= remaining_value:
    multiple = remaining_value / factor
    return stringify(multiple) + \
    _convert(remaining_value-multiple*factor, \
    base, exp-1)
    else:
    return "0" + \
    _convert(remaining_value, base, exp-1)
    else:
    return ""
    return "%s/%s" % (_convert(value, base, \
    int(math.log(value, base))), base)

  3. Add a docstring just below our function declaration that includes tests showing base 2 edges, base 36 edges, and the invalid base 37.

    def convert_to_basen(value, base):
    """Convert a base10 number to basen.
    These show the edges for base 2.
    >>> convert_to_basen(1, 2)
    '1/2'
    >>> convert_to_basen(2, 2)
    '10/2'
    >>> convert_to_basen(0, 2)
    Traceback (most recent call last):
    ...
    ValueError: math domain error
    These show the edges for base 36.
    >>> convert_to_basen(1, 36)
    '1/36'
    >>> convert_to_basen(35, 36)
    'z/36'
    >>> convert_to_basen(36, 36)
    '10/36'
    >>> convert_to_basen(0, 36)
    Traceback (most recent call last):
    ...
    ValueError: math domain error
    These show the edges for base 37.
    >>> convert_to_basen(1, 37)
    Traceback (most recent call last):
    ...
    Exception: Only support bases 2-36
    >>> convert_to_basen(36, 37)
    Traceback (most recent call last):
    ...
    Exception: Only support bases 2-36
    >>> convert_to_basen(37, 37)
    Traceback (most recent call last):
    ...
    Exception: Only support bases 2-36
    >>> convert_to_basen(0, 37)
    Traceback (most recent call last):
    ...
    Exception: Only support bases 2-36
    """
    if base < 2 or base > 36:

  4. Add a test runner.

    if __name__ == "__main__":
    import doctest
    doctest.testmod()

  5. Run the recipe.

    Python Testing Cookbook

How it works...

This version has a limit of handling base 2 through base 36.

For base 36, it uses a through z. This compared to base 16 using a through f. 35 in base 10 is represented as z in base 36.

We include several tests, including one for base 2 and base 36. We also test the maximum value before rolling over, and the next value, to show the rollover. For base 2, this is 1 and 2. For base 36, this is 35 and 36.

We have also included tests for 0 to show that our function doesn't handle this for any base. We also test base 37, which is invalid as well.

There's more...

It's important that our software works for valid inputs. It's just as important that our software works as expected for invalid inputs. We have documentation that can be viewed by our users when using our software that documents these edges. And thanks to Python's doctest module, we can test it and make sure that our software performs correctly.

 

Testing corner cases by iteration

Corner cases will appear as we continue to develop our code. By capturing corner cases in an iterable list, there is less code to write and capture another test scenario. This can increase our efficiency at testing new scenarios.

How to do it...

  1. Create a new file called recipe23.py and use it to store all our code for this recipe.
  2. Create a function that converts base 10 to any other base.

    def convert_to_basen(value, base):
    import math
    def _convert(remaining_value, base, exp):
    def stringify(value):
    if value > 9:
    return chr(value + ord('a')-10)
    else:
    return str(value)
    if remaining_value >= 0 and exp >= 0:
    factor = int(math.pow(base, exp))
    if factor <= remaining_value:
    multiple = remaining_value / factor
    return stringify(multiple) + \
    _convert(remaining_value-multiple*factor, \
    base, exp-1)
    else:
    return "0" + \
    _convert(remaining_value, base, exp-1)
    else:
    return ""
    return "%s/%s" % (_convert(value, base, \
    int(math.log(value, base))), base)

  3. Add some doc tests that include an array of input values to generate an array of expected outputs. Include one failure.

    def convert_to_basen(value, base):
    """Convert a base10 number to basen.
    Base 2
    >>> inputs = [(1,2,'1/2'), (2,2,'11/2')]
    >>> for value,base,expected in inputs:
    ... actual = convert_to_basen(value,base)
    ... assert actual == expected, 'expected: %s actual: %s' %
    (expected, actual)
    >>> convert_to_basen(0, 2)
    Traceback (most recent call last):
    ...
    ValueError: math domain error
    Base 36.
    >>> inputs = [(1,36,'1/36'), (35,36,'z/36'), (36,36,'10/36')]
    >>> for value,base,expected in inputs:
    ... actual = convert_to_basen(value,base)
    ... assert actual == expected, 'expected: %s actual: %s' %
    (expected, value)
    >>> convert_to_basen(0, 36)
    Traceback (most recent call last):
    ...
    ValueError: math domain error
    """
    import math

  4. Add a test runner.

    if __name__ == "__main__":
    import doctest
    doctest.testmod()

  5. Run the recipe.

    Python Testing Cookbook

In the previous screenshot, the key information is on this line: AssertionError: expected: 11/2 actual: 10/2. Is this test failure a bit contrived? Sure it is. But seeing a test case shows useful output is not. It's important to verify that our tests give us enough information to fix either the tests or the code.

How it works...

We created an array with each entry containing both the input data as well as the expected output. This provides us with an easy way to glance at a set of test cases.

Then, we iterate over each test case, calculate the actual value, and run it through a Python assert. An important part that is needed is the custom message 'expected: %s actual: %s'. Without it, we would never get the information to tell us which test case failed.

What if one test case fails?
If one of the tests in the array fails, then that code block exits and skips over the rest of the tests. This is the trade off for having a more succinct set of tests.

Does this type of test fit better into doctest or unittest?

Here are some criteria that are worth considering when deciding whether to put these tests in doctest:

  • Is the code easy to comprehend at a glance?
  • Is this clear, succinct, useful information when users view the docstrings?

If there is little value in having this in the documentation, and if it clutters the code, then that is a strong hint that this test block belongs to a separate test module.

 

Getting nosy with doctest

Up to this point, we have been either appending modules with a test runner, or we have typed python -m doctest <module> on the command line to exercise our tests.

For a quick recap, nose:

  • Provides us with the convenient test discovering tool nosetests
  • Is pluggable, with a huge ecosystem of available plugins
  • Includes a built-in plugin targeted at finding doctests and running them

Getting ready

We need to activate our virtual environment (virtualenv) and then install nose for this recipe.

  1. Create a virtual environment, activate it, and verify the tools are working.

    Python Testing Cookbook

  2. Using pip, install nose.

    Python Testing Cookbook

How to do it...

  1. Run nosetests –with-doctest against all the modules in this folder. If you notice, it prints a very short .....F.F...F, indicating that three tests have failed.
  2. Run nosetests –with-doctest -v to get a more verbose output. In the following screenshot, notice how the tests that failed. It is also valuable to see the <module>.<method> format with either ok or FAIL.

    Python Testing Cookbook

  3. Run nosetests –with-doctest against both the recipe19.py file as well as the recipe19 module, in different combinations.

    Python Testing Cookbook

How it works...

nosetests is targeted at discovering test cases and then running them. With this plugin, when it finds a docstring, it uses the doctest library to programmatically test it.

The doctest plugin is built around the assumption that doctests are not in the same package as other tests, like unittest. This means it will only run doctests found from non-test packages.

There isn't a whole lot of complexity in the nosetests tool, and...that's the idea!. In this recipe, we have seen how to use nosetests to get a hold of all the doctests.

Summary

In this article we saw ways to perform testing in Python using doctest.


Further resources on this subject:


Python Testing Cookbook Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools
Published: May 2011
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

About the Author :


Greg L. Turnquist

Greg has worked since 1997 as a software engineer at Harris Corporation, always seeking the right tool for the job. Since 2002, Greg has been part of the senior software team working on Harris' $3.5 billion FAA telco program, architecting mission-critical enterprise apps while managing a software team. He provides after hours support and 2nd-level engineering to support the nation-wide telco network and is no stranger to midnight failures and software triage. In 2010, Greg joined the SpringSource division of VMware.

Being a test-bitten script junky, Greg has used JUnit, TestNG, JMock, FEST, PyUnit, and pMock testing frameworks, along with other agile practices to produce top-quality code.

He has worked with Java, Spring, Spring Security, AspectJ, and Jython technologies and also developed sophisticated scripts for *nix and Windows platforms. Being a wiki evangelist, he also deployed a LAMP-based wiki website to provide finger-tip knowledge to users.

In 2006, Greg created the Spring Python project. The Spring Framework provided many useful features, and he wanted those same features available when working with Python.

Greg completed a master's degree in Computer Engineering at Auburn University, and lives in the United States with his family.

Books From Packt


Python Testing: Beginner's Guide
Python Testing: Beginner's Guide

Python Text Processing with NLTK 2.0 Cookbook
Python Text Processing with NLTK 2.0 Cookbook

Python Geospatial Development
Python Geospatial Development

Python 2.6 Graphics Cookbook
Python 2.6 Graphics Cookbookr

MySQL for Python
MySQL for Python

Python 3 Object Oriented Programming
Python 3 Object Oriented Programming

Spring Python 1.1
Spring Python 1.1

wxPython 2.8 Application Development Cookbook
wxPython 2.8 Application Development Cookbook


Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software