Testing and Debugging in Grok 1.0: Part 1

Exclusive offer: get 50% off this eBook here
Grok 1.0 Web Development

Grok 1.0 Web Development — Save 50%

Create flexible, agile web applications using the power of Grok—a Python web framework

$23.99    $12.00
by Carlos de la Guardia | February 2010 | Open Source Web Development

Some of the things that we will cover in this two-part article by Carlos de la Guardia, author of Grok 1.0 Web Development, includes:

  • Need for testing
  • Testing in Grok
  • Extending the functional test suite provided by grokproject
  • Other kinds of testing
  • Debugging tools

Grok offers some tools for testing, and in fact, a project created by grokproject (as the one we have been extending) includes a functional test suite. In this article, we are going to discuss testing a bit and then write some tests for the functionality that our application has so far.

Testing helps us avoid bugs, but it does not eliminate them completely, of course. There are times when we will have to dive into the code to find out what's going wrong. A good set of debugging aids becomes very valuable in this situation. We'll see that there are several ways of debugging a Grok application and also try out a couple of them.

Testing

It's important to understand that testing should not be treated as an afterthought.

As mentioned earlier, agile methodologies place a lot of emphasis on testing. In fact, there's even a methodology called Test Driven Development (TDD), which not only encourages writing tests for our code, but also writing tests before any other line of code.

There are various kinds of testing, but here we'll briefly describe only two:

  • Unit testing
  • Integration or functional tests

Unit testing

The idea of unit testing is to break a program into its constituent parts and test each one of them in isolation. Every method or function call can be tested separately to make sure that it returns the expected results and handles all of the possible inputs correctly.

An application which has unit tests that cover the majority of its lines of code, allows its developers to constantly run the tests after a change, and makes sure that modifications to the code do not break the existing functionality.

Functional tests

Functional tests are concerned with how the application behaves as a whole. In a web application, this means how it responds to a browser request and whether it returns the expected HTML for a given call.

Ideally, the customer himself has a hand in defining these tests, usually through explicit functionality requirements or acceptance criteria. The more formal the requirements from the customer are, the easier it is to define appropriate functional tests.

Testing in Grok

Grok highly encourages the use of both kinds of tests, and in fact, includes a powerful testing tool that is automatically configured with every project. In the Zope world—from where Grok originated—a lot of value is placed in a kind of tests known as "doctests", so Grok comes with a sample test suite of this kind.

Doctests

A doctest is a test that's written as a text file, with lines of code mixed with explanations of what the code is doing. The code is written in a way that simulates a Python interpreter session. As tests exercise large portions of the code (ideally 100%), they usually offer a good way of finding out of what an application does and how. So, if an application has no written documentation, its tests would be the next obvious way of finding out what it does. Doctests take this idea further by allowing the developer to explain in the text file exactly what each test is doing.

Doctests are especially useful for functional testing, because it makes more sense to document the high-level operations of a program. Unit tests, on the other hand, are expected to evaluate the program bit by bit and it can be cumbersome to write a text explanation for every little piece of code.

A possible drawback of doctests is that they can make the developer think that he needs no other documentation for his project. In almost all of the cases, this is not true. Documenting an application or package makes it immediately more accessible and useful, so it is strongly recommended that doctests should not be used as a replacement for good documentation. We'll show an example of using doctests in the Looking at the test code section of this article.

Default test setup for Grok projects

As mentioned above, Grok projects that are started with the grokproject tool already include a simple functional test suite by default. Let's examine it in detail.

Test configuration

The default test configuration looks for packages or modules that have the word 'tests' in their name and tries to run the tests inside. For functional tests, any files ending with .txt or .rst are considered.

For functional tests that need to simulate a browser, a special configuration is needed to tell Grok which packages to initialize in addition to the Grok infrastructure (usually the ones that are being worked on). The ftesting.zcml file in the package directory has this configuration. This also includes a couple of user definitions that are used by certain tests to examine functionality specific to a certain role, such as manager.

Test files

Besides the already mentioned ftesting.zcml file, in the same directory, there is a tests.py file added by grokproject, which basically loads the ZCML declarations and registers all of the tests in the package.

The actual tests that are included with the default project files are contained in the app.txt file. These are doctests that do a functional test run by loading the entire Grok environment and imitating a browser. We'll take a look at the contents of the file soon, but first let's run the tests.

Running the tests

As part of the project's build process, a script named test is included in the bin directory when you create a new project. This is the test runner and calling it without arguments, finds and executes all of the tests in the packages that are included in the configuration.

We haven't added a single test so far, so if we type bin/test in our project directory, we'll see more or less the same thing that doing that on a new project would show:

$ bin/test
Running tests at level 1
Running todo.FunctionalLayer tests:
Set up
in 12.319 seconds.
Running:
...2009-09-30 15:00:47,490 INFO sqlalchemy.engine.base.
Engine.0x...782c PRAGMA table_info("users")

2009-09-30 15:00:47,490 INFO sqlalchemy.engine.base.Engine.0x...782c ()

Ran 3 tests with 0 failures and 0 errors in 0.465 seconds.
Tearing down left over layers:
Tear down todo.FunctionalLayer ... not supported

The only difference between our output to that of a newly created Grok package is in the sqlalchemy lines. Of course, the most important part of the output is the "penultimate" line, which shows the number of tests that were run and whether there were any failures or errors. A failure means that some test didn't pass, which means that the code is not doing what it's supposed to do and needs to be checked. An error signifies that the code crashed unexpectedly at some point, and the test couldn't even be executed, so it's necessary to find the error and correct it before worrying about the tests.

The test runner

The test runner program looks for modules that contain tests. The test can be of three different types: Python tests, simple doctests, and full functionality doctests. To let the test runner know, which test file includes which kind of tests, a comment similar to the following is placed at the top of the file:

Do a Python test on the app.
:unittest:

In this case, the Python unit test layer will be used to run the tests. The other value that we are going to use is "doctest" when we learn how to write doctests.

The test runner then finds all of the test modules and runs them in the corresponding layer. Although unit tests are considered very important in regular development, we may find functional tests more necessary for a Grok web application, as we will usually be testing views and forms, which require the full Zope/Grok stack to be loaded to work. That's the reason why we find only functional doctests in the default setup.

Test layers

A test layer is a specific test setup which is used to differentiate the tests that are executed. By default, there is a test layer for each of the three types of tests handled by the test runner. It's possible to run a test layer without running the others and also to name new test layers to be able to cluster together tests that require a specific setup.

Invoking the test runner

As shown above, running bin/test will start the test runner with the default options. It's also possible to specify a number of options, and the most important ones are summarized below. In the following table, command-line options are shown to the left. Most options can be expressed with a short form (one dash) or a long form (two dashes). Arguments for the option in question are shown in uppercase.

-s PACKAGE,

--package=PACKAGE,

--dir=PACKAGE

Search the given package's directories for tests. This can be specified more than once, to run tests in multiple parts of the source tree. For example, when refactoring interfaces, you don't want to see the way you have broken setups for tests in other packages. You just want to run the interface tests. Packages are supplied as dotted names. For compatibility with the old test runner, forward and backward slashes in package names are converted to dots. (In the special case of packages, which are spread over multiple directories, only directories within the test search path are searched.)

-m MODULE,

--module=MODULE

Specify a test-module filter as a regular expression. This is a case sensitive regular expression, which is used in search (not match) mode, to limit which test modules are searched for tests. The regular expressions are checked against dotted module names. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The option can specy multiple test-module filters. Test modules matching any of the test filters are searched. If no test-module filter is specified, then all of the test modules are used.

-t TEST, --test=TEST

Specify a test filter as a regular expression. This is a case sensitive regular expression, which is used in search (not match) mode, to limit which tests are run. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The option can specify multiple test filters. Tests matching any of the test filters are included. If no test filter is specified, then all of the tests are executed.

--layer=LAYER

Specify a test layer to run. The option can be given multiple times to specify more than one layer. If not specified, all of the layers are executed. It is common for the running script to provide default values for this option. Layers are specified regular expressions that are used in search mode, for dotted names of objects that define a layer. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The layer named 'unit' is reserved for unit tests, however, take note of the -unit and non-unit options.

-u, --unit

Executes only unit tests, ignoring any layer options.

-f, --non-unit

Executes tests other than unit tests.

-v, --verbose

Makes output more verbose. Increment the verbosity level.

-q, --quiet

Makes the output minimal by overriding any verbosity options.

Looking at the test code

Let's take a look at the three default test files of a Grok project, to see what each one does.

ftesting.zcml

As we explained earlier, ftesting.zcml is a configuration file for the test runner. Its main objective is to help us set up the test instance with users, so that we can test different roles according to our needs.

<configure 
xmlns="http://namespaces.zope.org/zope"
i18n_domain="todo"
package="todo"
>

<include package="todo" />
<include package="todo_plus" />

<!-- Typical functional testing security setup -->
<securityPolicy
component="zope.securitypolicy.zopepolicy.ZopeSecurityPolicy"
/>

<unauthenticatedPrincipal
id="zope.anybody"
title="Unauthenticated User"
/>
<grant
permission="zope.View"
principal="zope.anybody"
/>

<principal
id="zope.mgr"
title="Manager"
login="mgr"
password="mgrpw"
/>

<role id="zope.Manager" title="Site Manager" />
<grantAll role="zope.Manager" />
<grant role="zope.Manager" principal="zope.mgr" />

As shown in the preceding code, the configuration simply includes a security policy, complete with users and roles and the packages that should be loaded by the instance, in addition to the regular Grok infrastructure. If we run any tests that require an authenticated user to work, we'll use these special users.

The includes at the top of the file just make sure that all of the Zope Component Architecture setup needed by our application is performed prior to running the tests.

tests.py

The default test module is very simple. It defines the functional layer and registers the tests for our package:

import os.path 
import z3c.testsetup
import todo
from zope.app.testing.functional import ZCMLLayer

ftesting_zcml = os.path.join(
os.path.dirname(todo.__file__), 'ftesting.zcml')
FunctionalLayer = ZCMLLayer(ftesting_zcml, __name__, 'FunctionalLayer', allow_teardown=True)

test_suite = z3c.testsetup.register_all_tests('todo')

After the imports, the first line gets the path for the ftesting.zcml file, which then is passed to the layer definition method ZCMLLayer. The final line in the module tells the test runner to find and register all of the tests in the package.

This will be enough for our testing needs in this article, but if we needed to create another non-Grok package for our application, we would need to add a line like the last one to it, so that all of its tests are found by the test runner. This is pretty much boilerplate code, as only the package name has to be changed.

Grok 1.0 Web Development Create flexible, agile web applications using the power of Grok—a Python web framework
Published: February 2010
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

app.txt

We finally come to the reason for this entire configuration—the actual tests that will be executed by the test runner. By default, the tests are included inside the app.txt file:

Do a functional doctest test on the app. 
========================================

:doctest:
:layer: todo.tests.FunctionalLayer

Let's first create an instance of Todo at the top level:

>>> from todo.app import Todo
>>> root = getRootFolder()
>>> root['app'] = Todo()

Run tests in the testbrowser
----------------------------

The zope.testbrowser.browser module exposes a Browser class that
simulates a web browser similar to Mozilla Firefox or IE. We use that to test how our application behaves in a browser. For more
information, see http://pypi.python.org/pypi/zope.testbrowser.

Create a browser and visit the instance you just created:

>>> from zope.testbrowser.testing import Browser
>>> browser = Browser()
>>> browser.open('http://localhost/app')

Check some basic information about the page you visit:

>>> browser.url
'http://localhost/app/@@login?camefrom=%2Fapp%2F%40%40index'
>>> browser.headers.get('Status').upper()
'200 OK'

The text file has a title and immediately after that a :doctest: declaration—a declaration which tells the test runner that these tests need a functional layer to be loaded for their execution. Then comes a :layer: declaration, which is a path that points to the layer that we defined earlier in tests.py. After that, comes the test code. Lines starting with three brackets represent the Python code that is tested. Anything else is commentary.

When using the Python interpreter, a line of code may return a value, in which case, the expected return value must be written immediately below that line. This expected value will be compared with the real return value of the tested code and a failure will be reported, if the values don't match. Similarly, a line which is followed by an empty line will produce a failure, when the code is executed and a result is returned, because it is assumed that the expected return value in that case is None.

For example, in the last line of the Python doctest, the expression browser.headers.get('Status').upper() is expected to return the value 200 OK. If anything else is returned, the test will fail, even if there's just a slight difference.

Adding our own tests

Now, let's add a few functional tests that are specific to our application. We will need to emulate a browser for that. The zope.testbrowser package includes a browser emulator. We can pass any valid URL to this browser by using browser.open, and it will send a request to our application exactly like a browser would. The response from our application will be then available as browser.contents, so that we can perform our testing comparisons on it.

The Browser class

Before writing our tests, it will be useful to see what exactly our testbrowser can do. Of course, anything that depends on JavaScript will not work here, but other than that, we can interact with links and even forms in a very straightforward manner. Here's a look at the main functionality offered by the Browser class:

Initialization

>>> from zope.testbrowser.testing import Browser

>>> browser = Browser()

Page contents

>>> print browser.contents

<html>

<head>

<title>Simple Page</title>

</head>

<body>

<h1>Simple Page</h1>

</body>

</html>

>>> '<h1>Simple Page</h1>' in browser.contents

True

Headers

>>> print browser.headers

Status: 200 OK

Content-Length: 123

Content-Type: text/html;charset=utf-8

X-Powered-By: Zope (www.zope.org), Python (www.python.org)

>>> browser.headers['content-type']

'text/html;charset=utf-8'

Cookies

>>> browser.cookies['foo']

'bar'

>>> browser.cookies.keys()

['foo']

>>> browser.cookies.values()

['bar']

>>> browser.cookies.items()

[('foo', 'bar')]

>>> 'foo' in browser.cookies

True

>>> browser.cookies['sha'] = 'zam'

Links

>>> link = browser.getLink('Link Text')

>>> link

<Link text='Link Text'

url='http://localhost/@@/testbrowser/navigate.html?message=By+Link+Text'>

>>> link.text

'Link Text'

>>> link.tag # links can also be image maps.

'a'

>>> link.url # it's normalized

'http://localhost/@@/testbrowser/navigate.html?message=By+Link+Text'

>>> link.attrs

{'href': 'navigate.html?message=By+Link+Text'}

>>> link.click()

>>> browser.url

'http://localhost/@@/testbrowser/navigate.html?message=By+Link+Text'

Form controls

>>> control = browser.getControl('Text Control')

>>> control

<Control name='text-value' type='text'>

>>> browser.getControl(label='Text Control') # equivalent

<Control name='text-value' type='text'>

>>> browser.getControl('Text Control').value = 'Other Text'

>>> browser.getControl('Submit').click()

 

Now that we know what we can do, let's try our hand at writing some tests.

Our first to-do application tests

Ideally, we should have been adding a couple of doctests to the app.txt file every time we add a new functionality to our application. We have gone through the reasons why we didn't do so, but let's recover some lost ground. At the very least, we'll get a feeling of how doctests work.

We'll add our new tests to the existing app.txt file. The last test there that we saw left us at the to-do instance URL. We are not logged in, so if we print the browser contents, we will get the login page. Let's add a test for this:

Since we haven't logged in, we can't see the application. The login
page appears:

>>> 'Username' in browser.contents
True
>>> 'Password' in browser.contents
True

As we mentioned earlier, when visiting a URL with the testbrowser, the entire HTML content of the page is stored in browser.contents. Now we know that our login page has a username and a password field, so we simply use a couple of in expressions and check if these fields evaluate to True. If they do, it would mean that the browser is effectively looking at the login page.

Let's add a test for logging in. When we start the application in the tests, the user database is empty, therefore, the most economical way of logging in is to use basic authentication. This can be easily done by changing the request headers:

 >>> browser.addHeader('Authorization', 'Basic mgr:mgrpw') 
>>> browser.open('http://localhost/app')
>>> 'Logged in as Manager' in browser.contents
True

That's it. We just add the header, "reload" the home page, and we should be logged in. We verify it by looking for the Logged in as message, which we know has to be there after a successful login.

Once we are logged in, we can finally test our application properly. Let's begin by adding a project:

We are now in. Let's create a project:

 >>> browser.getLink('Create a new project').click() 
>>> browser.getControl(name='form.title').value='a project'
>>> browser.getControl(name='form.description').value='The
description.'
>>> browser.getControl('Add project').click()
>>> browser.url
'http://localhost/app/0'
>>> 'Create new list' in browser.contents
True

First, we find the link on the home page that will take us to the 'add form' project. This is done easily with the help of the getLink method and the text of the link. We click on the link and now should have the form ready to fill in. We then use getControl to find each field by its name and change its value. Finally, we submit the form by getting the submit button control and clicking on it. The result is that the project is created and we are redirected to its main view. We can confirm this by comparing the browser url with the URL that we would expect in this case.

Adding a list to the project is just as easy. We get the form controls, assign them some values, and click on the submit button. The list and the link for adding new items to it should appear in the browser contents:

We have added a project. Now, we'll add a list to it. If we are successful, we will seea link for adding a new item for the list:

 >>> browser.getControl(name='title').value='a list' 
>>> browser.getControl(name='description').value='The list
description.'
>>> browser.getControl(name='new_list').click()
>>> 'New item' in browser.contents
True

Good. Let's see how we are doing so far:

$ bin/testRunning tests at level 1 
Running todo.FunctionalLayer tests:
Set up
in 3.087 seconds.
Running:
.......2009-09-30 21:35:44,585 INFO sqlalchemy.engine.base.
Engine.0x...69ec PRAGMA table_info("users")
2009-09-30 21:35:44,585 INFO sqlalchemy.engine.base.Engine.0x...69ec ()

Ran 7 tests with 0 failures and 0 errors in 0.428 seconds.
Tearing down left over layers:
Tear down todo.FunctionalLayer ... not supported


Not bad. We now have four more working tests than when we started.

Note that the test browser handles HTTP errors gracefully, returning a string similar to what a real browser would show when running into an error. For example, take a look at the following test:

>>> browser.open('http://localhost/invalid')
Traceback (most recent call last):
...
HTTPError: HTTP Error 404: Not Found

That's the default behavior because this is how real browsers work, but sometimes, when we are debugging, it's better to take a look at the original exception caused by our application. In such a case, we can make the browser stop handling errors automatically and throw the original exceptions, so that we can handle them. This is done by setting the browser.handleErrors property to False:

>>> browser.handleErrors = False
>>> browser.open('http://localhost/invalid')
Traceback (most recent call last):
...
NotFound: Object: <zope.site.folder.Folder object at ...>,
name: u'invalid'

>> Continue Reading Testing and Debugging in Grok 1.0: Part 2

 

[ 1 | 2 ]

If you have read this article you may be interested to view :

Grok 1.0 Web Development Create flexible, agile web applications using the power of Grok—a Python web framework
Published: February 2010
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

About the Author :


Carlos de la Guardia

Carlos has been doing web consulting and development since 1994, when selling any kind of project required two meetings just to explain what the Internet was in the first place. He was the co-founder of Aldea Systems, a web consulting company where he spent ten years working in all kinds of web projects using a diverse range of languages and tools. In 2005 he became an independent developer and consultant, specializing in Zope and Plone projects. He frequently blogs about Plone and other Zope-related subjects.

Books From Packt

Joomla! 1.5 Multimedia
Joomla! 1.5 Multimedia

TYPO3 4.3 Multimedia Cookbook
TYPO3 4.3 Multimedia Cookbook

jQuery 1.4 Reference Guide
jQuery 1.4 Reference Guide

Python Testing: Beginner's Guide
Python Testing: Beginner's Guide

Backbase 4 RIA Development
Backbase 4 RIA Development

AJAX and PHP: Building Modern Web Applications 2nd Edition
AJAX and PHP: Building Modern Web Applications 2nd Edition

jQuery UI 1.7: The User Interface Library for jQuery
    jQuery UI 1.7: The User Interface Library for jQuery

CodeIgniter 1.7
CodeIgniter 1.7

No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
q
e
h
6
w
U
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software