Python Testing: Coverage Analysis

Exclusive offer: get 50% off this eBook here
Python Testing Cookbook

Python Testing Cookbook — Save 50%

Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools

$26.99    $13.50
by Greg L. Turnquist | June 2011 | Open Source

Coverage analysis is measuring which lines in a program are run and which lines aren't. This type of analysis is also known as 'code coverage', or more simply 'coverage'.

In this article by Greg Lee Turnquist, author of Python Testing Cookbook, we will cover:

  • Building a network management application
  • Installing and running coverage on your test suite
  • Generating an HTML report using coverage
  • Generating an XML report using coverage
  • Getting nosy with coverage

 

Python Testing Cookbook

Python Testing Cookbook

Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools

        Read more about this book      

(For more resources on Python, see here.)

Introduction

A coverage analyzer can be used while running a system in production, but what are the pros and cons, if we used it this way? What about using a coverage analyzer when running test suites? What benefits would this approach provide compared to checking systems in production?

Coverage helps us to see if we are adequately testing our system. But it must be performed with a certain amount of skepticism. This is because, even if we achieve 100 percent coverage, meaning every line of our system was exercised, in no way does this guarantee us having no bugs. A quick example involves a code we write and what it processes is the return value from a system call. What if there are three possible values, but we only handle two of them? We may write two test cases covering our handling of it, and this could certainly achieve 100 percent statement coverage. However, it doesn't mean we have handled the third possible return value; thus, leaving us with a potentially undiscovered bug. 100 percent code coverage can also be obtained by condition coverage but may not be achieved with statement coverage. The kind of coverage we are planning to target should be clear.

Another key point is that not all testing is aimed at bug fixing. Another key purpose is to make sure that the application meets our customer's needs. This means that, even if we have 100 percent code coverage, we can't guarantee that we are covering all the scenarios expected by our users. This is the difference between 'building it right' and 'building the right thing'.

In this article, we will explore various recipes to build a network management application, run coverage tools, and harvest the results. We will discuss how coverage can introduce noise, and show us more than we need to know, as well as introduce performance issues when it instruments our code. We will also see how to trim out information we don't need to get a concise, targeted view of things.

This article uses several third-party tools in many recipes.

  • Spring Python (http://springpython.webfactional.com) contains many useful abstractions. The one used in this article is its DatabaseTemplate, which offers easy ways to write SQL queries and updates without having to deal with Python's verbose API. Install it by typing pip install springpython.

    Python Testing Cookbook

  • Install the coverage tool by typing pip install coverage. This may fail because other plugins may install an older version of coverage. If so, uninstall coverage by typing pip uninstall coverage, and then install it again with pip install coverage.
  • Nose is a useful test runner.

 

Building a network management application

For this article, we will build a very simple network management application, and then write different types of tests and check their coverage. This network management application is focused on digesting alarms, also referred to as network events. This is different from certain other network management tools that focus on gathering SNMP alarms from devices.

For reasons of simplicity, this correlation engine doesn't contain complex rules, but instead contains simple mapping of network events onto equipment and customer service inventory. We'll explore this in the next few paragraphs as we dig through the code.

How to do it...

With the following steps, we will build a simple network management application.

  1. Create a file called network.py to store the network application.
  2. Create a class definition to represent a network event.

    class Event(object):
    def __init__(self, hostname, condition, severity, event_time):
    self.hostname = hostname
    self.condition = condition
    self.severity = severity
    self.id = -1
    def __str__(self):
    return "(ID:%s) %s:%s - %s" % (self.id, self.hostname,
    self.condition, self.severity)

    • hostname: It is assumed that all network alarms originate from pieces of equipment that have a hostname.
    • condition: Indicates the type of alarm being generated. Two different alarming conditions can come from the same device.
    • severity: 1 indicates a clear, green status; and 5 indicates a faulty, red status.
    • id: The primary key value used when the event is stored in a database.
  3. Create a new file called network.sql to contain the SQL code.
  4. Create a SQL script that sets up the database and adds the definition for storing network events.

    CREATE TABLE EVENTS (
    ID INTEGER PRIMARY KEY,
    HOST_NAME TEXT,
    SEVERITY INTEGER,
    EVENT_CONDITION TEXT
    );

  5. Code a high-level algorithm where events are assessed for impact to equipment and customer services and add it to network.py.

    from springpython.database.core import*
    class EventCorrelator(object):
    def __init__(self, factory):
    self.dt = DatabaseTemplate(factory)
    def __del__(self):
    del(self.dt)
    def process(self, event):
    stored_event, is_active = self.store_event(event)
    affected_services, affected_equip = self.impact(event)
    updated_services = [
    self.update_service(service, event)
    for service in affected_services]
    updated_equipment = [
    self.update_equipment(equip, event)
    for equip in affected_equip]
    return (stored_event, is_active, updated_services,
    updated_equipment)

    The __init__ method contains some setup code to create a DatabaseTemplate. This is a Spring Python utility class used for database operations. See http://static.springsource.org/spring- python/1.2.x/sphinx/html/dao.html for more details. We are also using sqlite3 as our database engine, since it is a standard part of Python.

    The process method contains some simple steps to process an incoming event.

    • We first need to store the event in the EVENTS table. This includes evaluating whether or not it is an active event, meaning that it is actively impacting a piece of equipment.
    • Then we determine what equipment and what services the event impacts.
    • Next, we update the affected services by determining whether it causes any service outages or restorations.
    • Then we update the affected equipment by determining whether it fails or clears a device.
    • Finally, we return a tuple containing all the affected assets to support any screen interfaces that could be developed on top of this.
  6. Implement the store_event algorithm.

    def store_event(self, event):
    try:
    max_id = self.dt.query_for_int("""select max(ID)
    from EVENTS""")
    except DataAccessException, e:
    max_id = 0
    event.id = max_id+1
    self.dt.update("""insert into EVENTS
    (ID, HOST_NAME, SEVERITY,
    EVENT_CONDITION)
    values
    (?,?,?,?)""",
    (event.id, event.hostname,
    event.severity, event.condition))
    is_active = \
    self.add_or_remove_from_active_events(event)
    return (event, is_active)

    This method stores every event that is processed. This supports many things including data mining and post mortem analysis of outages. It is also the authoritative place where other event-related data can point back using a foreign key.

    • The store_event method looks up the maximum primary key value from the EVENTS table.
    • It increments it by one.
    • It assigns it to event.id.
    • It then inserts it into the EVENTS table.
    • Next, it calls a method to evaluate whether or not the event should be add to the list of active events, or if it clears out existing active events. Active events are events that are actively causing a piece of equipment to be unclear.
    • Finally, it returns a tuple containing the event and whether or not it was classified as an active event.

    For a more sophisticated system, some sort of partitioning solution needs to be implemented. Querying against a table containing millions of rows is very inefficient. However, this is for demonstration purposes only, so we will skip scaling as well as performance and security.

  7. Implement the method to evaluate whether to add or remove active events.

    def add_or_remove_from_active_events(self, event):
    """Active events are current ones that cause equipment
    and/or services to be down."""
    if event.severity == 1:
    self.dt.update("""delete from ACTIVE_EVENTS
    where EVENT_FK in (
    select ID
    from EVENTS
    where HOST_NAME = ?
    and EVENT_CONDITION = ?)""",
    (event.hostname,event.condition))
    return False
    else:
    self.dt.execute("""insert into ACTIVE_EVENTS
    (EVENT_FK) values (?)""",
    (event.id,))
    return True

    When a device fails, it sends a severity 5 event. This is an active event and in this method, a row is inserted into the ACTIVE_EVENTS table, with a foreign key pointing back to the EVENTS table. Then we return back True, indicating this is an active event.

  8. Add the table definition for ACTIVE_EVENTS to the SQL script.

    CREATE TABLE ACTIVE_EVENTS (
    ID INTEGER PRIMARY KEY,
    EVENT_FK,
    FOREIGN KEY(EVENT_FK) REFERENCES EVENTS(ID)
    );

    This table makes it easy to query what events are currently causing equipment failures.
    Later, when the failing condition on the device clears, it sends a severity 1 event. This means that severity 1 events are never active, since they aren't contributing to a piece of equipment being down. In our previous method, we search for any active events that have the same hostname and condition, and delete them. Then we return False, indicating this is not an active event.

  9. Write the method that evaluates the services and pieces of equipment that are affected by the network event.

    def impact(self, event):
    """Look up this event has impact on either equipment
    or services."""
    affected_equipment = self.dt.query(\
    """select * from EQUIPMENT
    where HOST_NAME = ?""",
    (event.hostname,),
    rowhandler=DictionaryRowMapper())
    affected_services = self.dt.query(\
    """select SERVICE.*
    from SERVICE
    join SERVICE_MAPPING SM
    on (SERVICE.ID = SM.SERVICE_FK)
    join EQUIPMENT
    on (SM.EQUIPMENT_FK = EQUIPMENT.ID
    where EQUIPMENT.HOST_NAME = ?""",
    (event.hostname,),
    rowhandler=DictionaryRowMapper())
    return (affected_services, affected_equipment)

    • We first query the EQUIPMENT table to see if event.hostname matches anything.
    • Next, we join the SERVICE table to the EQUIPMENT table through a many-to many relationship tracked by the SERVICE_MAPPING table. Any service that is related to the equipment that the event was reported on is captured.
    • Finally, we return a tuple containing both the list of equipment and list of services that are potentially impacted.

    Spring Python provides a convenient query operation that returns a list of objects mapped to every row of the query. It also provides an out-of-the-box DictionaryRowMapper that converts each row into a Python dictionary, with the keys matching the column names.

  10. Add the table definitions to the SQL script for EQUIPMENT, SERVICE, and SERVICE_MAPPING.

    CREATE TABLE EQUIPMENT (
    ID INTEGER PRIMARY KEY,
    HOST_NAME TEXT UNIQUE,
    STATUS INTEGER
    );
    CREATE TABLE SERVICE (
    ID INTEGER PRIMARY KEY,
    NAME TEXT UNIQUE,
    STATUS TEXT
    );
    CREATE TABLE SERVICE_MAPPING (
    ID INTEGER PRIMARY KEY,
    SERVICE_FK,
    EQUIPMENT_FK,
    FOREIGN KEY(SERVICE_FK) REFERENCES SERVICE(ID),
    FOREIGN KEY(EQUIPMENT_FK) REFERENCES EQUIPMENT(ID)
    );

  11. Write the update_service method that stores or clears service-related even and then updates the service's status based on the remaining active events.

    def update_service(self, service, event):
    if event.severity == 1:
    self.dt.update("""delete from SERVICE_EVENTS
    where EVENT_FK in (
    select ID
    from EVENTS
    where HOST_NAME = ?
    and EVENT_CONDITION = ?)""",
    (event.hostname,event.condition))
    else:
    self.dt.execute("""insert into SERVICE_EVENTS
    (EVENT_FK, SERVICE_FK)
    values (?,?)""",
    (event.id,service["ID"]))
    try:
    max = self.dt.query_for_int(\
    """select max(EVENTS.SEVERITY)
    from SERVICE_EVENTS SE
    join EVENTS
    on (EVENTS.ID = SE.EVENT_FK)
    join SERVICE
    on (SERVICE.ID = SE.SERVICE_FK)
    where SERVICE.NAME = ?""",
    (service["NAME"],))
    except DataAccessException, e:
    max = 1
    if max > 1 and service["STATUS"] == "Operational":
    service["STATUS"] = "Outage"
    self.dt.update("""update SERVICE
    set STATUS = ?
    where ID = ?""",
    (service["STATUS"], service["ID"]))
    if max == 1 and service["STATUS"] == "Outage":
    service["STATUS"] = "Operational"
    self.dt.update("""update SERVICE
    set STATUS = ?
    where ID = ?""",
    (service["STATUS"], service["ID"]))
    if event.severity == 1:
    return {"service":service, "is_active":False}
    else:
    return {"service":service, "is_active":True}

    Service-related events are active events related to a service. A single event can be related to many services. For example, what if we were monitoring a wireless router that provided Internet service to a lot of users, and it reported a critical error? This one event would be mapped as an impact to all the end users. When a new active event is processed, it is stored in SERVICE_EVENTS for each related service.
    Then, when a clearing event is processed, the previous service event must be deleted from the SERVICE_EVENTS table.

  12. Add the table defnition for SERVICE_EVENTS to the SQL script.

    CREATE TABLE SERVICE_EVENTS (
    ID INTEGER PRIMARY KEY,
    SERVICE_FK,
    EVENT_FK,
    FOREIGN KEY(SERVICE_FK) REFERENCES SERVICE(ID),
    FOREIGN KEY(EVENT_FK) REFERENCES EVENTS(ID)
    );

    It is important to recognize that deleting an entry from SERVICE_EVENTS doesn't mean that we delete the original event from the EVENTS table. Instead, we are merely indicating that the original active event is no longer active and it does not impact the related service.

  13. Prepend the entire SQL script with drop statements, making it possible to run the script for several recipes
    DROP TABLE IF EXISTS SERVICE_MAPPING;
    DROP TABLE IF EXISTS SERVICE_EVENTS;
    DROP TABLE IF EXISTS ACTIVE_EVENTS;
    DROP TABLE IF EXISTS EQUIPMENT;
    DROP TABLE IF EXISTS SERVICE;
    DROP TABLE IF EXISTS EVENTS;
  14. Append the SQL script used for database setup with inserts to preload some equipment and services.
    INSERT into EQUIPMENT (ID, HOST_NAME, STATUS) values (1, 
    'pyhost1', 1);
    INSERT into EQUIPMENT (ID, HOST_NAME, STATUS) values (2,
    'pyhost2', 1);
    INSERT into EQUIPMENT (ID, HOST_NAME, STATUS) values (3,
    'pyhost3', 1);
    INSERT into SERVICE (ID, NAME, STATUS) values (1, 'service-abc',
    'Operational');
    INSERT into SERVICE (ID, NAME, STATUS) values (2, 'service-xyz',
    'Outage');
    INSERT into SERVICE_MAPPING (SERVICE_FK, EQUIPMENT_FK) values
    (1,1);
    INSERT into SERVICE_MAPPING (SERVICE_FK, EQUIPMENT_FK) values
    (1,2);
    INSERT into SERVICE_MAPPING (SERVICE_FK, EQUIPMENT_FK) values
    (2,1);
    INSERT into SERVICE_MAPPING (SERVICE_FK, EQUIPMENT_FK) values
    (2,3);
  15. Finally, write the method that updates equipment status based on the current active events.

    def update_equipment(self, equip, event):
    try:
    max = self.dt.query_for_int(\
    """select max(EVENTS.SEVERITY)
    from ACTIVE_EVENTS AE
    join EVENTS
    on (EVENTS.ID = AE.EVENT_FK)
    where EVENTS.HOST_NAME = ?""",
    (event.hostname,))
    except DataAccessException:
    max = 1
    if max != equip["STATUS"]:
    equip["STATUS"] = max
    self.dt.update("""update EQUIPMENT
    set STATUS = ?""",
    (equip["STATUS"],))
    return equip

Here, we need to find the maximum severity from the list of active events for a given host name. If there are no active events, then Spring Python raises a DataAccessException and we translate that to a severity of 1.

We check if this is different from the existing device's status. If so, we issue a SQL update. Finally, we return the record for the device, with its status updated appropriately.

How it works...

This application uses a database-backed mechanism to process incoming network events, and checks them against the inventory of equipment and services to evaluate failures and restorations. Our application doesn't handle specialized devices or unusual types of services. This real-world complexity has been traded in for a relatively simple application, which can be used to write various test recipes.

Events typically map to a single piece of equipment and to zero or more services. A service can be thought of as a string of equipment used to provide a type of service to the customer. New failing events are considered active until a clearing event arrives. Active events, when aggregated against a piece of equipment, define its current status. Active events, when aggregated against a service, defines the service's current status.

 

Python Testing Cookbook Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools
Published: May 2011
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

 

        Read more about this book      

(For more resources on Python, see here.)

Installing and running coverage on your test suite

Install the coverage tool and run it against your test suite. Then you can view a report showing what lines were covered by the test suite.

How to do it...

With the following steps, we will build some unit tests and then run them through the coverage tool.

  1. Create a new file called recipe52.py to contain our test code for this recipe.
  2. Write a simple unit test that injects a single, alarming event into the system.

    from network import *
    import unittest
    from springpython.database.factory import *
    from springpython.database.core import *
    class EventCorrelationTest(unittest.TestCase):
    def setUp(self):
    db_name = "recipe52.db"
    factory = Sqlite3ConnectionFactory(db_name)
    self.correlator = EventCorrelator(factory)
    dt = DatabaseTemplate(factory)
    sql = open("network.sql").read().split(";")
    for statement in sql:
    dt.execute(statement + ";")
    def test_process_events(self):
    evt1 = Event("pyhost1", "serverRestart", 5)
    stored_event, is_active, \
    updated_services, updated_equipment = \
    self.correlator.process(evt1)
    print "Stored event: %s" % stored_event
    if is_active:
    print "This event was an active event."
    print "Updated services: %s" % updated_services
    print "Updated equipment: %s" % updated_equipment
    print "---------------------------------"
    if __name__ == "__main__":
    unittest.main()

  3. Clear out any existing coverage report data using coverage -e.
  4. Run the test suite using the coverage tool.
    gturnquist$ coverage -x recipe52.py
    Stored event: (ID:1) pyhost1:serverRestart - 5
    This event was an active event.
    Updated services: [{'is_active': True, 'service': {'STATUS':
    'Outage', 'ID': 1, 'NAME': u'service-abc'}}, {'is_active': True,
    'service': {'STATUS': u'Outage', 'ID': 2, 'NAME': u'service-
    xyz'}}]
    Updated equipment: [{'STATUS': 5, 'ID': 1, 'HOST_NAME':
    u'pyhost1'}]
    ---------------------------------
    .
    --------------------------------------------------
    ----
    Ran 1 test in 0.211s
    OK
  5. Print out the report captured by the previous command by typing coverage -r. If the report shows several other modules listed from Python's standard libraries, it's a hint that you have an older version of the coverage tool installed. If so, uninstall the old version by typing pip uninstall coverage followed by reinstalling with pip install coverage.

    (Move the mouse over the image to enlarge.)

  6. Create another file called recipe52b.py to contain a different test suite.
  7. Write another test suite that generates two faults and then clears them out.

    from network import *
    import unittest
    from springpython.database.factory import *
    from springpython.database.core import *
    class EventCorrelationTest(unittest.TestCase):
    def setUp(self):
    db_name = "recipe52b.db"
    factory = Sqlite3ConnectionFactory(db=db_name)
    self.correlator = EventCorrelator(factory)
    dt = DatabaseTemplate(factory)
    sql = open("network.sql").read().split(";")
    for statement in sql:
    dt.execute(statement + ";")
    def test_process_events(self):
    evt1 = Event("pyhost1", "serverRestart", 5)
    evt2 = Event("pyhost2", "lineStatus", 5)
    evt3 = Event("pyhost2", "lineStatus", 1)
    evt4 = Event("pyhost1", "serverRestart", 1)
    for event in [evt1, evt2, evt3, evt4]:
    stored_event, is_active, \
    updated_services, updated_equipment = \
    self.correlator.process(event)
    print "Stored event: %s" % stored_event
    if is_active:
    print "This event was an active event."

    print "Updated services: %s" % updated_services
    print "Updated equipment: %s" % updated_equipment
    print "---------------------------------"
    if __name__ == "__main__":
    unittest.main()

  8. Run this test suite through the coverage tool using coverage -x recipe52b.py.
  9. Print out the report by typing coverage -r.

The first test suite only injects a single alarm. We expect it to cause a service outage as well as taking its related piece of equipment down. Since this would not exercise any of the event clearing logic, we certainly don't expect 100 percent code coverage.

In the report, we can see it scoring network.py as having 65 statements, and having executed 55 of them, resulting in 85 percent coverage. We also see that recipe52.py had 23 statements and executed all of them. This means all of our test code ran.

At this point, we realize that we are only testing the alarming part of the event correlator. To make this more effective, we should inject another alarm followed by a couple of clears to make sure that everything clears out and the services return to operational status. This should result in 100 percent coverage in our simple application.

The second screenshot indeed shows that we have reached full coverage of network.py.

There's more...

We also see Spring Python reported as well. If we had used any other third-party libraries, then they would also appear. Is this right? It depends. The previous comments seem to indicate that we don't really care about coverage of Spring Python but, in other situations, we may be very interested. And how can the coverage tool know where to draw the line?

Why are there no asserts in the unit test?

It is true that the unit test isn't adequate with regard to testing the outcome. To draw up this recipe, I visually inspected the output to see whether the network management application was performing as expected. But this is incomplete. A real production grade unit test needs to finish this with a set of assertions so that visual scanning is not needed.

So why didn't we code any? Because the focus of this recipe was on how to generate a coverage report and then use that information to enhance the testing. We covered both of those. By thinking about what was and wasn't tested, we wrote a comprehensive test that shows services going into outage and back to operational status. We just didn't put the finishing touch of confirming this automatically.

 

Generating an HTML report using coverage

Using the coverage tool, generate an HTML visual coverage report. This is useful because we can drill into the source code and see what lines were not exercised in the test procedures.

Reading a coverage report without reading the source code is not very useful. It may be tempting to compare two different projects based on the coverage percentages. But unless the actual code is analyzed, this type of comparison can lead to faulty conclusions about the quality of software.

How to do it...

With these steps, we will explore creating a nicely viewable HTML coverage report.

  1. Generate coverage metrics by following the steps in Installing and running coverage on your test suite recipe and only running the first test suite (which has resulted in less than 100 percent coverage).
  2. Generate an HTML report by typing: coverage.html.
  3. Open htmlcov/index.html using your favorite browser and inspect the overall report.

    Python Testing Cookbook

  4. Click on network, and scroll down to see where the event clearing logic wasn't exercised due to no clearing events being processed.

    Python Testing Cookbook

How it works...

The coverage tool has a built-in feature to generate an HTML report. This provides a powerful way to visually inspect the source code and see what lines were not executed.

By looking at this report, we can clearly see that the lines not executed involve the lack of clearing network events that are being processed. This can tip us off about another test case which involves clearing events that need to be drafted.

 

Generating an XML report using coverage

The coverage tool can generate an XML coverage report in Cobertura format ( http://cobertura.sourceforge.net/). This is useful if we want to process the coverage information in another tool. In this recipe, we will see how to use the coverage command-line tool, and then view the XML report by hand.

It's important to understand that reading a coverage report without reading the source code is not very useful. It may be tempting to compare two different projects based on the coverage percentages. But unless the actual code is analyzed, this type of comparison can lead to faulty conclusions about the quality of the software.

For example, a project with 85 percent coverage may appear, on the surface, to be better tested than one with 60 percent. However, if the 60 percent application has much more thoroughly exhaustive scenarios - as they are only covering the core parts of the system that are in heavy use - then it may be much more stable than the 85 percent application.

Coverage analysis serves a useful purpose when comparing test results between iterations, and using it to decide which scenarios need to be added to our testing repertoire.

How to do it...

With these steps, we will discover how to create an XML report using the coverage tool, consumable by other tools:

  1. Generate coverage metrics.
  2. Generate an XML report by typing: coverage xml.
  3. Open coverage.xml using your favorite text or XML editor. The format of the XML is the same as Cobertura-a Java code coverage analyzer. This means that many tools, like Jenkins, can parse the results.

How it works...

The coverage tool has a built-in feature to generate an XML report. This provides a powerful way to parse the output using some type of external tool.

In the previous screenshot, I opened it using SpringSource Tool Suite (you can download it from http://www.springsource.com/developer/sts), partly because I happen to use STS every day, but you can use any text or XML editor you like.

What use is an XML report?

XML is not the best way to communicate coverage information to users. Generating an HTML report with coverage is a more practical recipe when it comes to human users.

What if we want to capture a coverage report and publish it inside a continuous integration system like Jenkins? All we need to do is install the Cobertura plugin (refer https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin), and this report becomes traceable. Jenkins can nicely monitor trends in coverage and give us more feedback as we develop our system.

 

Getting nosy with coverage

Install the coverage nose plugin, and run your test suite using nose. This provides a quick and convenient report using the ubiquitous nosetests tool. This recipe assumes you have already created the network management application as described in the Building a network management application section.

How to do it...

With these steps, we will see how to combine the coverage tool with nose.

  1. Create a new file called recipe55.py to store our test code.
  2. Create a test case that injects a faulting alarm.

    from network import *
    import unittest
    from springpython.database.factory import *
    from springpython.database.core import *
    class EventCorrelationTest(unittest.TestCase):
    def setUp(self):
    db_name = "recipe55.db"
    factory = Sqlite3ConnectionFactory(db=db_name)
    self.correlator = EventCorrelator(factory)
    dt = DatabaseTemplate(factory)
    sql = open("network.sql").read().split(";")
    for statement in sql:
    dt.execute(statement + ";")
    def test_process_events(self):
    evt1 = Event("pyhost1", "serverRestart", 5)
    stored_event, is_active, \
    updated_services, updated_equipment = \
    self.correlator.process(evt1)
    print "Stored event: %s" % stored_event
    if is_active:
    print "This event was an active event."
    print "Updated services: %s" % updated_services
    print "Updated equipment: %s" % updated_equipment
    print "---------------------------------"

  3. Run the test module using the coverage plugin by typing nosetests recipe55 - with-coverage.

Python Testing Cookbook

How it works...

The nose plugin for coverage invokes the coverage tool and provides a formatted report. For each module, it displays:

  • Total number of statements
  • Number of missed statements
  • Percentage of covered statements
  • Line numbers for the missed statements

There's more...

A common behavior of nose is to alter stdout, disabling the print statements embedded in the test case.

Why use the nose plugin instead of the coverage tool directly?

The coverage tool works fine by itself, as was demonstrated in other recipes in this article, however, nose is a ubiquitous testing tool used by many developers. Providing a plugin makes it easy to support this vast community by empowering them to run the exact set of test plugins they want, with coverage being part of that test complement.

Why are sqlite3 and springpython included?

Sqlite3 is a relational database library that is included with Python. It is file based which means that no separate processes are required to create and use a database. Details about Spring Python can be found in the earlier sections of this article.

The purpose of this recipe was to measure coverage of our network management application and the corresponding test case. So why are these third-party libraries included? The coverage tool has no way of automatically knowing what we want and the things we don't want to see from a coverage perspective.

Summary

In this article we saw how to create coverage reports and interpret them correctly. We also saw how to tie them in with your continuous integration system.


Further resources on this subject:


Python Testing Cookbook Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools
Published: May 2011
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

About the Author :


Greg L. Turnquist

Greg has worked since 1997 as a software engineer at Harris Corporation, always seeking the right tool for the job. Since 2002, Greg has been part of the senior software team working on Harris' $3.5 billion FAA telco program, architecting mission-critical enterprise apps while managing a software team. He provides after hours support and 2nd-level engineering to support the nation-wide telco network and is no stranger to midnight failures and software triage. In 2010, Greg joined the SpringSource division of VMware.

Being a test-bitten script junky, Greg has used JUnit, TestNG, JMock, FEST, PyUnit, and pMock testing frameworks, along with other agile practices to produce top-quality code.

He has worked with Java, Spring, Spring Security, AspectJ, and Jython technologies and also developed sophisticated scripts for *nix and Windows platforms. Being a wiki evangelist, he also deployed a LAMP-based wiki website to provide finger-tip knowledge to users.

In 2006, Greg created the Spring Python project. The Spring Framework provided many useful features, and he wanted those same features available when working with Python.

Greg completed a master's degree in Computer Engineering at Auburn University, and lives in the United States with his family.

Books From Packt


Python Testing: Beginner's Guide
Python Testing: Beginner's Guide

Python Text Processing with NLTK 2.0 Cookbook
Python Text Processing with NLTK 2.0 Cookbook

Python Geospatial Development
Python Geospatial Development

Python 2.6 Graphics Cookbook
Python 2.6 Graphics Cookbookr

MySQL for Python
MySQL for Python

Python 3 Object Oriented Programming
Python 3 Object Oriented Programming

Spring Python 1.1
Spring Python 1.1

wxPython 2.8 Application Development Cookbook
wxPython 2.8 Application Development Cookbook


Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software