Django Debugging Overview

Exclusive offer: get 50% off this eBook here
Django 1.1 Testing and Debugging

Django 1.1 Testing and Debugging — Save 50%

Building rigorously tested and bug-free Django applications

$29.99    $15.00
by Karen M. Tracey | April 2010 | Content Management Open Source Web Development

This article by Karen M. Tracey, author of the book Django 1.1 Testing and Debugging, introduces Django's debugging support. Specifically, this article will:

  • List the Django settings that control the collection and presentation of debugging information, and briefly describe the effects of enabling debug
  • Illustrate the results of running with debug enabled in the case of serious code failure
  • Describe the database query history that is collected with debug enabled, and show how to access it

Django debug settings

Django has a number of settings that control the collection and presentation of debug information. The primary one is named DEBUG; it broadly controls whether the server operates in development (if DEBUG is True) or production mode.

In development mode, the end-user is expected to be a site developer. Thus, if an error arises during processing of a request, it is useful to include specific technical information about the error in the response sent to the web browser. This is not useful in production mode, when the user is expected to be simply a general site user.

This section describes three Django settings that are useful for debugging during development. Additional settings are used during production to control what errors should be reported, and where error reports should be sent. These additional settings will be discussed in the section on handling problems in production.

The DEBUG and TEMPLATE_DEBUG settings

DEBUG is the main debug setting. One of the most obvious effects of setting this to True is that Django will generate fancy error page responses in the case of serious code problems, such as exceptions raised during processing of a request. If TEMPLATE_DEBUG is also True, and the exception raised is related to a template error, then the fancy error page will also include information about where in the template the error occurred.

The default value for both of these settings is False, but the settings.py file created by manage.py startproject turns both of them on by including these lines at the top of the file:

DEBUG = True
TEMPLATE_DEBUG = DEBUG

Note that setting TEMPLATE_DEBUG to True when DEBUG is False isn't useful. The additional information collected with TEMPLATE_DEBUG turned on will never be displayed if the fancy error pages, controlled by the DEBUG setting, are not displayed. Similarly, setting TEMPLATE_DEBUG to False when DEBUG is True isn't very useful. In this case, for template errors, the fancy debug page will be lacking helpful information. Thus, it makes sense to keep these settings tied to each other, as previously shown.

Details on the fancy error pages and when they are generated will be covered in the next section. Besides generating these special pages, turning DEBUG on has several other effects. Specifically, when DEBUG is on:

  • A record is kept of all queries sent to the database. Details of what is recorded and how to access it will be covered in a subsequent section.
  • For the MySQL database backend, warnings issued by the database will be turned into Python Exceptions. These MySQL warnings may indicate a serious problem, but a warning (which only results in a message printed to stderr) may pass unnoticed. Since most development is done with DEBUG turned on, raising exceptions for MySQL warnings then ensures that the developer is aware of the possible issue.
  • The admin application performs extensive validation of the configuration of all registered models and raises an ImproperlyConfigured exception on the first attempt to access any admin page if an error is found in the configuration. This extensive validation is fairly expensive and not something you'd generally want done during production server start-up, when the admin configuration likely has not changed since the last start-up. When running with DEBUG on, though, it is possible that the admin configuration has changed, and thus it is useful and worth the cost to do the explicit validation and provide a specific error message about what is wrong if a problem is detected.
  • Finally, there are several places in Django code where an error will occur while DEBUG is on, and the generated response will contain specific information about the cause of the error, whereas when DEBUG is off the generated response will be a generic error page.

The TEMPLATE_STRING_IF_INVALID setting

A third setting that can be useful for debugging during development is TEMPLATE_STRING_IF_INVALID. The default value for this setting is the empty string. This setting is used to control what gets inserted into a template in place of a reference to an invalid (for example, non-existent in the template context) variable. The default value of an empty string results in nothing visible taking the place of such invalid references, which can make them hard to notice. Setting TEMPLATE_STRING_IF_INVALID to some value can make tracking down such invalid references easier.

However, some code that ships with Django (the admin application, in particular), relies on the default behavior of invalid references being replaced with an empty string. Running code like this with a non-empty TEMPLATE_STRING_IF_INVALID setting can produce unexpected results, so this setting is only useful when you are specifically trying to track down something like a misspelled template variable in code that always ensures that variables, even empty ones, are set in the template context.

Debug error pages

With DEBUG on, Django generates fancy debug error pages in two circumstances:

  • When a django.http.Http404 exception is raised
  • When any other exception is raised and not handled by the regular view processing code

In the latter case, the debug page contains a tremendous amount of information about the error, the request that caused it, and the environment at the time it occurred. The debug pages for Http404 exceptions are considerably simpler.

To see examples of the Http404 debug pages, consider the survey_detail view

def survey_detail(request, pk):
survey = get_object_or_404(Survey, pk=pk)
today = datetime.date.today()
if survey.closes < today:
return display_completed_survey(request, survey)
elif survey.opens > today:
raise Http404
else:
return display_active_survey(request, survey)

There are two cases where this view may raise an Http404 exception: when the requested survey is not found in the database, and when it is found but has not yet opened. Thus, we can see the debug 404 page by attempting to access the survey detail for a survey that does not exist, say survey number 24. The result will be as follows:

Notice there is a message in the middle of the page that describes the cause of the page not found response: No Survey matches the given query. This message was generated automatically by the get_object_or_404 function. By contrast, the bare raise Http404 in the case where the survey is found but not yet open does not look like it will have any descriptive message. To confirm this, add a survey that has an opens date in the future, and try to access its detail page. The result will be something like the following:

That is not a very helpful debug page, since it lacks any information about what was being searched for and why it could not be displayed. To make this page more useful, include a message when raising the Http404 exception. For example:

raise Http404("%s does not open until %s; it is only %s" %
(survey.title, survey.opens, today))

Then an attempt to access this page will be a little more helpful:

Note that the error message supplied with the Http404 exception is only displayed on the debug 404 page; it would not appear on a standard 404 page. So you can make such messages as descriptive as you like and not worry that they will leak private or sensitive information to general users.

Another thing to note is that a debug 404 page is only generated when an Http404 exception is raised. If you manually construct an HttpResponse with a 404 status code, it will be returned, not the debug 404 page. Consider this code:

return HttpResponse("%s does not open until %s; it is only %s" %
(survey.title, survey.opens, today), status=404)

If that code were used in place of the raise Http404 variant, then the browser will simply display the passed message:

Without the prominent Page not found message and distinctive error page formatting, this page isn't even obviously an error report. Note also that some browsers by default will replace the server-provided content with a supposedly "friendly" error page that tends to be even less informative. Thus, it is both easier and more useful to use the Http404 exception instead of manually building HttpResponse objects with status code 404.

A final example of the debug 404 page that is very useful is the one that is generated when URL resolution fails. For example, if we add an extra space before the survey number in the URL, the debug 404 page generated will be as follows:

The message on this page includes all of the information necessary to figure out why URL resolution failed. It includes the current URL, the name of the base URLConf used for resolution, and all patterns that were tried, in order, for matching.

If you do any significant amount of Django application programming, it's highly likely that at some time this page will appear and you will be convinced that one of the listed patterns should match the given URL. You would be wrong. Do not waste energy trying to figure out how Django could be so broken. Rather, trust the error message, and focus your energies on figuring out why the pattern you think should match doesn't in fact match. Look carefully at each element of the pattern and compare it to the actual element in the current URL: there will be something that doesn't match.

In this case, you might think the third listed pattern should match the current URL. The first element in the pattern is the capture of the primary key value, and the actual URL value does contain a number that could be a primary key. However, the capture is done using the pattern \d+. An attempt to match this against the actual URL characters—a space followed by 2—fails because \d only matches numeric digits and the space character is not a numeric digit. There will always be something like this to explain why the URL resolution failed.

For now, we will leave the subject of debug pages and learn about accessing the history of database queries that is maintained when DEBUG is on.

Django 1.1 Testing and Debugging Building rigorously tested and bug-free Django applications
Published: April 2010
eBook Price: $29.99
Book Price: $49.99
See more
Select your format and quantity:

Database query history

When DEBUG is True, Django maintains a history of all SQL commands sent to the database. This history is kept in a list, named queries, located in the django. db.connection module. The easiest way to see what is kept in this list is to examine it from a shell session:

>>> from django.db import connection
>>> connection.queries
[]
>>> from survey.models import Survey
>>> Survey.objects.count()
2
>>> connection.queries
[{'time': '0.002', 'sql': u'SELECT COUNT(*) FROM "survey_survey"'}]
>>>

Here we see that queries is initially empty at the beginning of the shell session. We then retrieve a count of the number of Survey objects in the database, which comes back as 2. When we again display the contents of queries, we see that there is now one query in the queries list. Each element in the list is a dictionary containing two keys: time and sql. The value of time is how long, in seconds, the query took to execute. The value of sql is the actual SQL query that was sent to the database.

One thing to note about the SQL contained in connection.queries: it does not include quoting of query parameters. For example, consider the SQL shown for a query on Surveys with titles that start with Christmas:

>>> Survey.objects.filter(title__startswith='Christmas')
[<Survey: Christmas Wish List (opens 2009-11-26, closes 2009-12-31)>]
>>> print connection.queries[-1]['sql']
SELECT "survey_survey"."id", "survey_survey"."title",
"survey_survey"."opens", "survey_survey"."closes"
FROM "survey_survey" WHERE "survey_survey"."title"
LIKE Christmas% ESCAPE '\' LIMIT 21
>>>

In the displayed SQL, Christmas% would need to be quoted in order for the SQL to be valid. However, we see here it is not quoted when stored in connection. queries. The reason is because Django does not actually pass the query in this form to the database backend. Rather, Django passes parameterized queries. That is, the passed query string contains parameter placeholders, and parameter values are passed separately. It is up to the database backend, then, to perform parameter substitution and proper quoting.

For the debug information placed in connection.queries, Django does parameter substitution, but it does not attempt to do the quoting, as that varies from backend to backend. So do not be concerned by the lack of parameter quoting in connection. queries: it does not imply that parameters are not quoted correctly when they are actually sent to the database. It does mean, though, that the SQL from connection. queries cannot be successfully cut and pasted directly into a database shell program. If you want to use the SQL form connection.queries in a database shell, you will need to supply the missing parameter quoting.

You might have noticed and may be curious about the LIMIT 21 included in the previous SQL. The QuerySet requested did not include a limit, so why did the SQL include a limit? This is a feature of the QuerySet repr method, which is what the Python shell calls to display the value returned by the Survey.objects.filter call.

A QuerySet may have many elements, and displaying the entire set, if it is quite large, is not particularly useful in Python shell sessions, for example. Therefore, QuerySet repr displays a maximum of 20 items. If there are more, repr will add an ellipsis to the end to indicate that the display is incomplete. Thus, the SQL resulting from a call to repr on a QuerySet will limit the result to 21 items, which is enough to determine if an ellipsis is needed to indicate that the printed result is incomplete.

Any time you see LIMIT 21 included in a database query, that is a signal the query was likely the result of a call to repr. Since repr is not frequently called from application code, such queries are likely resulting from other code (such as the Python shell, here, or a graphical debugger variable display window) that may be automatically displaying the value of a QuerySet variable. Keeping this in mind can help reduce confusion when trying to figure out why some queries are appearing in connection.queries.

There is one final item to note about connection.queries: despite the name, it is not limited to just SQL queries. All SQL statements sent to the database, including updates and inserts, are stored in connection.queries. For example, if we create a new Survey from the shell session, we will see the resulting SQL INSERT stored in connection.queries:

>>> import datetime
>>> Survey.objects.create(title='Football Favorites',opens=datetime.date.
today())
<Survey: Football Favorites (opens 2009-09-24, closes 2009-10-01)>
>>> print connection.queries[-1]['sql']
INSERT INTO "survey_survey" ("title", "opens", "closes") VALUES (Football
Favorites, 2009-09-24, 2009-10-01)
>>>

Here we have been accessing connection.queries from a shell session. Often, however, it may be useful to see what it contains after a request has been processed. That is, we might want to know what database traffic was generated during the creation of a page. Recreating the calling of a view function from within a Python shell and then manually examining connection.queries is not particularly convenient, however. Therefore, Django provides a context processor, django.core. contextprocessors.debug, that provides convenient access to the data stored in connection.queries from a template.

Summary

We have now completed the overview of debugging support in Django. Specifically, we have:

  • Learned about the Django settings that control the collection and presentation of debug information
  • Seen how when debug is turned on, special error pages are produced that help with the task of debugging problems
  • Learned about the history of database queries that is maintained when debugging is turned on, and saw how to access it

If you have read this article you may be interested to view :

Django 1.1 Testing and Debugging Building rigorously tested and bug-free Django applications
Published: April 2010
eBook Price: $29.99
Book Price: $49.99
See more
Select your format and quantity:

About the Author :


Karen M. Tracey

    Karen has a PhD in Electrical/Computer Engineering from the University of Notre Dame. Her research there focused on distributed operating systems, which led to work in industry centered on communications protocols and middleware. Outside of work she has an interest in puzzles, which led her to take up crossword construction. She has published nearly 100 puzzles in The New York Times, The Los Angeles Times syndicate, The New York Sun, and USA Today. She amassed a database of thousands of puzzles to aid in constructing and cluing her own puzzles. The desire to put a web frontend on this database is what led her to Django. She was impressed by the framework and its community, and became an active core framework contributor. Karen is one of the most prolific posters on the Django users mailing list. Her experience in helping hundreds of people there guided her in choosing the best and most useful material to include in this book.            

Books From Packt

CMS Made Simple 1.6: Beginner's Guide
CMS Made Simple 1.6: Beginner's Guide

Joomla! 1.5: Beginner's Guide
Joomla! 1.5: Beginner's Guide

TYPO3 4.2 E-Commerce
TYPO3 4.2 E-Commerce

CodeIgniter 1.7 Professional Development
CodeIgniter 1.7 Professional Development

Zabbix 1.8 Network Monitoring
Zabbix 1.8 Network Monitoring

Magento 1.3 Sales Tactics Cookbook
Magento 1.3 Sales Tactics Cookbook

Apache MyFaces 1.2 Web Application Development
Apache MyFaces 1.2 Web Application Development

Drupal 7 First look
Drupal 7 First look

No votes yet
I just received my copy for by
I just received my copy for review in the mail last night! I'm excited to dig in and learn everything there is about testing and debugging in Django!

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
i
r
h
d
6
z
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software