You might be a programmer, a coder, a developer, or maybe a hacker. As such, it's almost impossible that you haven't had to sit down with a program that you were sure was ready for use—or possibly a program you knew was not ready—and put together a bunch of tests to prove it. It often feels like an exercise in futility or, at its best, a waste of time. We're going to learn about how to avoid this situation, and make testing an easy and enjoyable process.
This book is going to show you a new way to test, a way that puts much of the burden of testing right where it should be—on the computer. Even better, your tests will help you to find problems early, and tell you just where they are, so that you can fix them easily. You'll love the easy, helpful methods of automated testing, and test-driven development.
The Python language has some of the best tools when it comes to testing, so we're going to learn about how to make testing easy, quick, fun, and productive by taking advantage of these tools.
This chapter provides an overview of the book, so we're going to briefly discuss the following topics:
The levels of tests: Unit, integration, and system
Acceptance testing and regression testing
Test-driven development
This chapter started with a lot of grandiose claims—you'll enjoy testing. You'll rely on this to help you kill bugs early and easily. Testing will stop being a burden for you and will become something that you want to do. How?
Think back to the last really annoying bug that you had to deal with. It could have been anything: a database schema mismatch, a bad data structure, what have you.
Remember what caused the bug? The one line of code with the subtle logic error. The function that didn't do what the docs said it did. Whatever it was, keep this in mind.
Imagine a small chunk of code that could have caught that bug, if it had been run at the right time and you had been told about it.
Now imagine that all of your code is accompanied by those little chunks of test code, and that they are quick and easy to execute.
How long would your bug have survived? Not very long at all.
This gives you a pretty basic understanding of what we'll be talking about in this book. There are many refinements and tools to make the process quicker and easier, but the basic idea is to tell the computer what you expect, using simple and easily-written chunks of code, and then tell the computer to double-check your expectations throughout the coding process. Because expectations are easy to describe, you can write them down first, allowing the computer to shoulder much of the burden of debugging your code. Because expectations are easy to describe, you can write them down fast, allowing you to move on to interesting things while the computer keeps track of the rest.
When you're done, you have a code base that is highly tested and that you can be highly confident of. You caught your bugs early and fixed them quickly. Best of all, your testing was done by the computer based on what you told it and what you wanted the program to do. After all, why should you do it, when the computer can do it for you?
I have had simple automated tests catch everything from minor typos to instances of database access code being left dangerously out-of-date after a schema change, and pretty much any other bug that you can imagine. The tests caught the errors quickly and pinpointed their locations. A great deal of effort and trouble was avoided because they were there.
Spending less time on debugging and being sure of your result makes programming more fun. Producing a higher quality of code in a shorter amount of time makes it more profitable. The test suite provides instant feedback, allowing you to run each chunk of your code now instead of waiting for the program as a whole to be in a state where you can execute it. This quick turnaround makes programming both more satisfying and more productive.
Testing is commonly divided into several categories based on how complex the component being tested is. Most of our time will be focused on the lowest level—unit testing—because unit tests provide the foundation for tests in the other categories. Tests in the other categories operate on the same principles.
Unit testing is testing the smallest possible pieces of a program. Often, this means individual functions or methods. The keyword here is individual: something is a "unit" if there's no meaningful way to divide it up further.
For example, it would make sense in order to consider this function as a unit:
def quadratic(a, b, c, x): return a * (x ** 2) + b * x + c
The preceding function works as a unit because breaking it up into smaller pieces is not something that can be practically or usefully done.
Unit tests test a single unit in isolation, verifying that it works as expected without considering what the rest of the program might do. This protects each unit from inheriting bugs from the mistakes made elsewhere, and makes it easy to narrow down where the real problems are.
By itself, unit testing isn't enough to confirm that a complete program works correctly, but it's the foundation on which everything else is based. You can't build a house without solid materials, and you can't build a program without units that work as expected.
In integration testing, the boundaries of isolation are pushed further back, so that the tests encompass the interactions between related units. Each test should still be run in isolation in order to avoid inheriting problems from outside, but now the test checks whether the tested units behave correctly as a group.
Integration testing can be performed with the same tools as unit testing. For this reason, newcomers to automated testing are sometimes lured into ignoring the distinction between unit testing and integration testing. Ignoring this distinction is dangerous because such multipurpose tests often make assumptions about the correctness of some of the units they involve; this means that the tester loses much of the benefit that automated testing would have granted. We're not aware of the assumptions we make until they bite us, so we need to consciously choose to work in a way that minimizes assumptions. That's one of the reasons why I refer to test-driven development as a "discipline."
System testing extends the boundaries of isolation even further to the point where they don't even exist. System tests check parts of the program after the whole thing has been plugged together. In a sense, system tests are an extreme form of the integration tests.
System tests are very important, but they're not very useful without the integration tests and unit tests backing them up. You have to be sure of the pieces before you can be sure of the whole. If there's a subtle error somewhere, system testing will tell you that it exists, but not where it is or how to fix it. The odds are good that you've experienced this situation before; it's probably why you hate testing. With a properly put together test suite, system tests are almost a formality. Most of the problems are caught by unit tests or integration tests, while the system tests simply provide reassurance that all is well.
When a program is first specified, we decide what behavior is expected out of it. Tests that are written to confirm that the program actually does what was expected are called acceptance tests. Acceptance tests can be written at any of the previously discussed levels, but most often you will see them at the integration or system level.
Acceptance tests tend to be the exception to the rule about progressing from unit tests to integration tests and then to system tests. Many program specifications describe the program at a fairly high level, and acceptance tests need to operate at the same level as that of the specification. It's not uncommon for the majority of system tests to be acceptance tests.
Acceptance tests are nice to have because they provide you with ongoing assurance that the program you're creating is actually the program that was specified.
A regression is when a part of your code that once functioned correctly stops doing so. Most often, that is a result of changes made elsewhere in the code undermining the assumptions of the now-buggy section. When this happens, it's a good idea to add tests to your test suite that can recognize the bug. This ensures that, if you ever make a similar mistake again, the test suite will catch it immediately.
Tests that make sure that the working code doesn't become buggy are called regression tests. They can be written before or after a bug is found, and they provide you with the assurance that your program's complexity is not causing the bugs to multiply. Once your code passes a unit test, integration test, or a system test, you don't need to delete these tests from the test suite. You can leave them in place, and they will function as additional regression tests, letting you know if the test stops working.
When you combine all of the elements we've introduced in this chapter, you will arrive at the discipline of test-driven development. In test-driven development, you always write the tests first. Once you have tests for the code you're about to write, and only then, you will write the code that makes the tests pass.
This means that the first thing you will do is write the acceptance tests. Then you figure out which units of the program you're going to start with, and write tests—nominally, these are the regression tests, even though the bug they're catching at first is "the code doesn't exist"; this confirms that these units are not yet functioning correctly. Then you can write some code that makes the unit-level regression tests pass.
The process continues until the whole program is complete: write tests, then write code that makes the tests pass. If you discover a bug that isn't caught by an existing test, add a test first, then add or modify the code to make the test pass. The end result is a very solid program, thanks to all the bugs that were caught early, easily, and in less time.
This book assumes that you have a working knowledge of the Python programming language, specifically, Version 3.4 or higher of that language. If you don't have Python already, you can download the complete language toolkit and library from http://www.python.org/, as a single easily-installed package.
Tip
Most versions of Linux and Mac OS X already include Python, but not necessarily a new version that will work with this book. Run Python from the command line to check.
You'll also require your favorite text editor, preferably one that has language support for Python. Popular choices for editors include emacs, Vim, Geany, gedit, and Notepad++. For those willing to pay, TextMate and Sublime are popular.
Note
Some of these popular editors are somewhat... exotic. They have their own operating idiom, and don't behave like any other program you might have used. They're popular because they are highly functional; they may be weird, though. If you find that one editor doesn't suit you, just pick a different one.
In this chapter, we learned about what you can expect to learn from this book as well as talking a little bit about the philosophy of automated testing and test-driven development.
We talked about the different levels and roles of tests that combine to form a complete suite of tests for a program: unit tests, integration tests, system tests, acceptance tests, and regression tests. We learned that unit tests are the tests of the fundamental components of a program (such as functions); integration tests are the tests that cover larger swathes of a program (such as modules); system tests are the tests that cover a program in its entirety; acceptance tests make sure that the program is what it's supposed to be; and regression tests ensure that it keeps working as we develop it.
We talked about how automated testing can help you by moving the burden of testing mostly onto the computer. You can tell the computer how to check your code, instead of having to do the checks yourself. This makes it convenient to check your code early and more often, saves you from overlooking things you would otherwise miss, and helps you to quickly locate and fix bugs.
We talked about test-driven development, the discipline of writing your tests first, and letting them tell you what needs to be done in order to write the code you need. We also briefly discussed the development environment you'll require in order to work through this book.
Now, we're ready to move on to working with the doctest testing tool, the subject of the next chapter.
