Python: Using doctest for Documentation

Exclusive offer: get 50% off this eBook here
Python Testing Cookbook

Python Testing Cookbook — Save 50%

Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools

£16.99    £8.50
by Greg L. Turnquist | May 2011 | Open Source

Python provides the useful ability to embed comments inside functions that are accessible from a Python shell. These are known as docstrings. A docstring provides the ability to embed not only information, but also code samples that are runnable.In this article, we will explore different ways to use doctest to develop documentation.

In this article by Greg Lee Turnquist, author of Python Testing Cookbook, we will cover:

  • Documenting the basics
  • Catching stack traces
  • Running doctests from the command line
  • Printing out all your documentation including a status report

 

Python Testing Cookbook

Python Testing Cookbook

Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools

        Read more about this book      

(For more resources on Python, see here.)

The reader can benefit from the previous article on Testing in Python using doctest.

 

Documenting the basics

Python provides out-of-the-box capability to put comments in code known as docstrings. Docstrings can be read when looking at the source and also when inspecting the code interactively from a Python shell. In this recipe, we will demonstrate how these interactive docstrings can be used as runnable tests.

What does this provide? It offers easy-to-read code samples for the users. Not only are the code samples readable, they are also runnable, meaning we can ensure the documentation stays up to date.

How to do it...

With the following steps, we will create an application combined with runnable docstring comments, and see how to execute these tests:

  1. Create a new file named recipe16.py to contain all the code we write for this recipe.
  2. Create a function that converts base-10 numbers to any other base using recursion.

    def convert_to_basen(value, base):
    import math
    def _convert(remaining_value, base, exp):
    def stringify(value):
    if value > 9:
    return chr(value + ord('a')-10)
    else:
    return str(value)
    if remaining_value >= 0 and exp >= 0:
    factor = int(math.pow(base, exp))
    if factor <= remaining_value:
    multiple = remaining_value / factor
    return stringify(multiple) + \
    _convert(remaining_value-multiple*factor, \
    base, exp-1)
    else:
    return "0" + \
    _convert(remaining_value, base, exp-1)
    else:
    return ""

    return "%s/%s" % (_convert(value, base, \
    int(math.log(value, base))), base)

  3. Add a docstring just below the external function, as shown in the highlighted section of the following code. This docstring declaration includes several examples of using the function.

    def convert_to_basen(value, base):
    """Convert a base10 number to basen.edur
    >>> convert_to_basen(1, 2)
    '1/2'
    >>> convert_to_basen(2, 2)
    '10/2'
    >>> convert_to_basen(3, 2)
    '11/2'
    >>> convert_to_basen(4, 2)
    '100/2'
    >>> convert_to_basen(5, 2)
    '101/2'
    >>> convert_to_basen(6, 2)
    '110/2'
    >>> convert_to_basen(7, 2)
    '111/2'
    >>> convert_to_basen(1, 16)
    '1/16'
    >>> convert_to_basen(10, 16)
    'a/16'
    >>> convert_to_basen(15, 16)
    'f/16'
    >>> convert_to_basen(16, 16)
    '10/16'
    >>> convert_to_basen(31, 16)
    '1f/16'
    >>> convert_to_basen(32, 16)
    '20/16'
    """
    import math

  4. Add a test runner block that invokes Python's doctest module.

     

    if __name__ == "__main__":
    import doctest
    doctest.testmod()

     

  5. From an interactive Python shell, import the recipe and view its documentation.

    Python Testing Cookbook

  6. Run the code from the command line. In the next screenshot, notice how nothing is printed. This is what happens when all the tests pass.

    Python Testing Cookbook

  7. Run the code from the command line with -v to increase verbosity. In the following screenshot, we see a piece of the output, showing what was run and what was expected. This can be useful when debugging doctest.

    Python Testing Cookbook

How it works...

The doctest module looks for blocks of Python inside docstrings and runs it like real code. >>> is the same prompt we see when we use the interactive Python shell. The following line shows the expected output. doctest runs the statements it sees and then compares the actual with the expected output.

There's more...

doctest is very picky when matching expected output with actual results.

  • An extraneous space or tab can cause things to break.
  • Structures like dictionaries are tricky to test, because Python doesn't guarantee the order of items. On each test run, the items could be stored in a different order. Simply printing out a dictionary is bound to break it.
  • It is strongly advised not to include object references in expected outputs. These values also vary every time the test is run.

 

Catching stack traces

It's a common fallacy to write tests only for successful code paths. We also need to code against error conditions including the ones that generate stack traces. With this recipe, we will explore how stack traces are pattern-matched in doc testing that allows us to confirm expected errors.

How to do it...

With the following steps, we will see how to use doctest to verify error conditions:

  1. Create a new file called recipe17.py to write all our code for this recipe.
  2. Create a function that converts base 10 numbers to any other base using recursion.

    def convert_to_basen(value, base):
    import math
    def _convert(remaining_value, base, exp):
    def stringify(value):
    if value > 9:
    return chr(value + ord('a')-10)
    else:
    return str(value)
    if remaining_value >= 0 and exp >= 0:
    factor = int(math.pow(base, exp))
    if factor <= remaining_value:
    multiple = remaining_value / factor
    return stringify(multiple) + \
    _convert(remaining_value-multiple*factor, \
    base, exp-1)
    else:
    return "0" + \
    _convert(remaining_value, base, exp-1)
    else:
    return ""
    return "%s/%s" % (_convert(value, base, \
    int(math.log(value, base))), base)

  3. Add a docstring just below the external function declaration that includes two examples that are expected to generate stack traces.

    def convert_to_basen(value, base):
    """Convert a base10 number to basen.
    >>> convert_to_basen(0, 2)
    Traceback (most recent call last):
    ...
    ValueError: math domain error
    >>> convert_to_basen(-1, 2)
    Traceback (most recent call last):
    ...
    ValueError: math domain error
    """
    import math

  4. Add a test runner block that invokes Python's doctest module.

    if __name__ == "__main__":
    import doctest
    doctest.testmod()

  5. Run the code from the command line. In the following screenshot, notice how nothing is printed. This is what happens when all the tests pass.

    Python Testing Cookbook

  6. Run the code from the command line with -v to increase verbosity. In the next screenshot, we can see that 0 and -1 generate math domain errors. This is due to using math.log to find the starting exponent.

    Python Testing Cookbook

How it works...

The doctest module looks for blocks of Python inside docstrings and runs it like real code. >>>; is the same prompt we see when we use the interactive Python shell. The following line shows the expected output. doctest runs the statements it sees and then compares the actual output with the expected output.

With regard to stack traces, there is a lot of detailed information provided in the stack trace. Pattern matching the entire trace is ineffective. By using the ellipsis, we are able to skip the intermediate parts of the stack trace and just match on the distinguishing part: ValueError: math domain error.

This is valuable, because our users can see not only the way it handles good values, but will also observe what errors to expect when bad values are provided.

 

Python Testing Cookbook Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools
Published: May 2011
eBook Price: £16.99
Book Price: £27.99
See more
Select your format and quantity:

 

        Read more about this book      

(For more resources on Python, see here.)

 

Running doctests from the command line

We have seen how to develop tests by embedding runnable fragments of code in docstrings. But for each of these tests we had to make the module runnable. What if we wanted to run something other than our doctests from the command line? We would have to get rid of the doctest.testmod() statements!

The good news is that starting with Python 2.6, there is a command-line option to run a specific module using doctest without coding a runner.
Typing: python -m doctest -v example.py will import example.py and run it through doctest.testmod(). According to documentation, this may fail if the module is part of a package and imports other submodules.

How to do it...

In the following steps, we will create a simple application. We will add some doctests and then run them from the command line without writing a special test runner.

  1. Create a new file called recipe18.py to store the code written for this recipe.
  2. Create a function that converts base 10 numbers to any other base using recursion.

    def convert_to_basen(value, base):
    import math
    def _convert(remaining_value, base, exp):
    def stringify(value):
    if value > 9:
    return chr(value + ord('a')-10)
    else:
    return str(value)
    if remaining_value >= 0 and exp >= 0:
    factor = int(math.pow(base, exp))
    if factor <= remaining_value:
    multiple = remaining_value / factor
    return stringify(multiple) + \
    _convert(remaining_value-multiple*factor, \
    base, exp-1)
    else:
    return "0" + \
    _convert(remaining_value, base, exp-1)
    else:
    return ""
    return "%s/%s" % (_convert(value, base, \
    int(math.log(value, base))), base)

  3. Add a docstring just below the external function declaration that includes some of the tests.

    def convert_to_basen(value, base):
    """Convert a base10 number to basen.
    >>> convert_to_basen(10, 2)
    '1010/2'
    >>> convert_to_basen(15, 16)
    'f/16'
    >>> convert_to_basen(0, 2)
    Traceback (most recent call last):
    ...
    ValueError: math domain error
    >>> convert_to_basen(-1, 2)
    Traceback (most recent call last):
    ...
    ValueError: math domain error
    """
    import math

  4. Run the code from the command line using -m doctest. As shown in the following screenshot, no output indicates that all the tests have passed.

    Python Testing Cookbook

  5. Run the code from the command line with -v to increase verbosity. What happens if we forget to include -m doctest? Using the -v option helps us to avoid this by giving us a warm fuzzy that our tests are working.

    Python Testing Cookbook

How it works...

We use the __main__ block of a module to run other test suites. What if we wanted to do the same here? We would have to pick whether __main__ would be for unittest tests, doctests, or both! What if we didn't even want to run testing through __main__, but instead run our application?

That is why Python added the option of invoking testing right from the command line using -m doctest.

Don't you want to know for sure if your tests are running or, whether they are working? Is the test suite really doing what it promised? With other tools, we usually have to embed print statements, or deliberate failures, just to know things are being trapped properly. Doesn't doctest's -v option provide a convenient quick glance at what's happening?

 

Printing out all your documentation including a status report

Let's build a script that takes a set of modules and prints out a complete report, showing all documentation as well as running any given tests.

This is a valuable recipe, because it shows us how to use Python's APIs to harvest a code-driven runnable report. This means the documentation is accurate and up to date, reflecting the current state of our code.

How to do it...

In the following steps, we will write an application and some doctests. Then we will build a script to harvest a useful report.

  1. Create a new file called recipe21_report.py to contain the script that harvests our report.
  2. Start creating a script by importing Python's inspect library as the basis for drilling down into a module from inspect import*
  3. Add a function that focuses on either printing out an item's __doc__ string or prints out no documentation found.

    def print_doc(name, item):
    if item.__doc__:
    print "Documentation for %s" % name
    print "-------------------------------"
    print item.__doc__
    print "-------------------------------"
    else:
    print "Documentation for %s - None" % name

  4. Add a function that prints out the documentation based on a given module. Make sure this function looks for classes, methods, and functions, and prints out their docs.

    def print_docstrings(m, prefix=""):
    print_doc(prefix + "module %s" % m.__name__, m)
    for (name, value) in getmembers(m, isclass):
    if name == '__class__': continue
    print_docstrings(value, prefix=name + ".")
    for (name, value) in getmembers(m, ismethod):
    print_doc("%s%s()" % (prefix, name), value)
    for (name, value) in getmembers(m, isfunction):
    print_doc("%s%s()" % (prefix, name), value)

  5. Add a runner that parses the command-line string, and iterates over each provided module.

    if __name__ == "__main__":
    import sys
    import doctest
    for arg in sys.argv[1:]:
    if arg.startswith("-"): continue
    print "==============================="
    print "== Processing module %s" % arg
    print "==============================="
    m = __import__(arg)
    print_docstrings(m)
    print "Running doctests for %s" % arg
    print "-------------------------------"
    doctest.testmod(m)

  6. Create a new file called recipe21.py to contain an application with tests that we will run the earlier script against.
  7. In recipe21.py, create a shopping cart app and fill it with docstrings and doctests.

    """ This is documentation for the entire recipe.
    With it, we can demonstrate usage of the code.
    >>> cart = ShoppingCart().add("tuna sandwich", 15.0)
    >>> len(cart)
    1
    >>> cart.item(1)
    'tuna sandwich'
    >>> cart.price(1)
    15.0
    >>> print round(cart.total(9.25), 2)
    16.39
    """
    class ShoppingCart(object):
    """
    This object is used to store the goods.
    It conveniently calculates total cost including
    tax.
    """
    def __init__(self):
    self.items = []
    def add(self, item, price):
    "Add an item to the internal list."
    self.items.append(Item(item, price))
    return self
    def item(self, index):
    "Look up the item. The cart is a 1-based index."
    return self.items[index-1].item
    def price(self, index):
    "Look up the price. The cart is a 1-based index."
    return self.items[index-1].price
    def total(self, sales_tax):
    "Add up all costs, and then apply a sales tax."
    sum_price = sum([item.price for item in self.items])
    return sum_price*(1.0 + sales_tax/100.0)
    def __len__(self):
    "Support len(cart) operation."
    return len(self.items)
    class Item(object):
    def __init__(self, item, price):
    self.item = item
    self.price = price

  8. Run the report script against this module using -v, and look at the screen's output.
    ===============================
    == Processing module recipe21
    ===============================
    Documentation for module recipe21
    -------------------------------

    This is documentation for the this entire recipe.
    With it, we can demonstrate usage of the code.
    >>> cart = ShoppingCart().add("tuna sandwich", 15.0)
    >>> len(cart)
    1
    >>> cart.item(1)
    'tuna sandwich'
    >>> cart.price(1)
    15.0
    >>> print round(cart.total(9.25), 2)
    16.39
    -------------------------------
    Documentation for Item.module Item - None
    Documentation for Item.__init__() - None
    Documentation for ShoppingCart.module ShoppingCart
    -------------------------------
    This object is used to store the goods.
    It conveniently calculates total cost including
    tax.
    ...
    Running doctests for recipe21
    -------------------------------
    Trying:
    cart = ShoppingCart().add("tuna sandwich", 15.0)
    Expecting nothing
    ok
    Trying:
    len(cart)
    Expecting:
    1
    ok
    5 tests in 10 items.
    5 passed and 0 failed.
    Test passed.

How it works...

This script is tiny, yet harvests a lot of useful information.

By using Python's standard inspect module, we are able to drill down starting at the module level. The reflective way to look up a docstring is by accessing the __doc__ property of an object. This is contained in modules, classes, methods, and functions. They exist in other places, but we limited our focus for this recipe.

We ran it in verbose mode, to show that the tests were actually executed. We hand parsed the command-line options, but doctest automatically looks for -v to decide whether or not to turn on verbose output. To prevent our module processor from catching this and trying to process it as another module, we added a line to skip any -xyz style flags.

if arg.startswith("-"): continue

There's more...

We could spend more time enhancing this script. For example, we could dump this out with an HTML markup, making it viewable in a web browser. We could also find third party libraries to export it in other ways.

We could also work on refining where it looks for docstrings and how it handles them. In our case, we just printed them to the screen. A more reusable approach would be to return some type of structure containing all the information. Then the caller can decide whether to print to screen, encode it in HTML, or generate a PDF document.

This isn't necessary, however, because this recipe's focus is on seeing how to mix these powerful out-of-the-box options which Python provides into a quick and useful tool.

Summary

In this article we took at the different ways we can use doctest to develop documentation.


Further resources on this subject:


Python Testing Cookbook Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools
Published: May 2011
eBook Price: £16.99
Book Price: £27.99
See more
Select your format and quantity:

About the Author :


Greg L. Turnquist

Greg has worked since 1997 as a software engineer at Harris Corporation, always seeking the right tool for the job. Since 2002, Greg has been part of the senior software team working on Harris' $3.5 billion FAA telco program, architecting mission-critical enterprise apps while managing a software team. He provides after hours support and 2nd-level engineering to support the nation-wide telco network and is no stranger to midnight failures and software triage. In 2010, Greg joined the SpringSource division of VMware.

Being a test-bitten script junky, Greg has used JUnit, TestNG, JMock, FEST, PyUnit, and pMock testing frameworks, along with other agile practices to produce top-quality code.

He has worked with Java, Spring, Spring Security, AspectJ, and Jython technologies and also developed sophisticated scripts for *nix and Windows platforms. Being a wiki evangelist, he also deployed a LAMP-based wiki website to provide finger-tip knowledge to users.

In 2006, Greg created the Spring Python project. The Spring Framework provided many useful features, and he wanted those same features available when working with Python.

Greg completed a master's degree in Computer Engineering at Auburn University, and lives in the United States with his family.

Books From Packt


Python Testing: Beginner's Guide
Python Testing: Beginner's Guide

Python Text Processing with NLTK 2.0 Cookbook
Python Text Processing with NLTK 2.0 Cookbook

Python Geospatial Development
Python Geospatial Development

Python 2.6 Graphics Cookbook
Python 2.6 Graphics Cookbookr

MySQL for Python
MySQL for Python

Python 3 Object Oriented Programming
Python 3 Object Oriented Programming

Spring Python 1.1
Spring Python 1.1

wxPython 2.8 Application Development Cookbook
wxPython 2.8 Application Development Cookbook


Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software