Chapter 1. Setting Up a Test Infrastructure

Modern web development is witnessing a JavaScript renaissance, with the expanding popularity of frontend-driven, single-page, and real-time web applications. Leading and facilitating the charge are a number of JavaScript web frameworks that enable developers to sensibly organize frontend web applications into modular and convention-driven components. As more logic and functionality is pushed from the server to the browser, these frameworks are increasingly critical in maintaining single-page application state, avoiding unstructured and ad hoc "spaghetti" code, and providing abstractions and functionality for commonly encountered development situations.

This book will focus on one such framework—Backbone.js (http://backbonejs.org/)—that stands out from the crowd with a well-balanced feature set including small footprint size, solid core abstractions, and significant community support. Backbone.js provides a minimum set of useful interfaces (for example, models, collections, routers, and views) for application development while maintaining an enormous amount of flexibility with pluggable template engines, extensible events for cross-component communication, and a generally agnostic approach to code interaction and patterns. The framework is used at scale in applications for organizations such as USA Today, LinkedIn, Hulu, Foursquare, Disqus, and many others. Essentially, Backbone.js provides practical tools for data-driven, client-heavy web application development without getting too much in the way.

However, this evolving world of frontend development is scattered with many potential stumbling blocks. More specifically, while the theoretical application possibilities with modern JavaScript frameworks such as Backbone.js are endless, one of the most critical issues looming over rapid application development in this sphere is software quality and reliability.

JavaScript web applications are already notoriously difficult to verify and test: asynchronous DOM events and data requests are subject to timing issues and spurious failures, display behavior is difficult to isolate from application logic, and test suites depend on/interact with a specific browser. Frontend frameworks such as Backbone.js add another level of complexity with additional interfaces that need to be isolated and tested, large numbers of various small components interacting concurrently, and event logic propagating throughout application layers. Moreover, the implementation agnostic paradigm of Backbone.js produces wildly varying application code bases, making test guidelines and heuristics something of a moving target.

In this book, we will tackle the challenge of testing Backbone.js applications by identifying the parts of an application to be tested, asserting correct behavior of various components, and verifying that the program works as intended as an integrated whole. Kicking things off in this chapter, we will introduce a basic test infrastructure in the following parts:

Designing a repository structure in which to develop Backbone.js applications and tests
Getting the Mocha, Chai, and Sinon.JS test libraries
Setting up and writing our first tests
Running and assessing test results with the Mocha test reporter

We assume that the reader is already comfortable with JavaScript web application development and familiar with Backbone.js and its usual complements—Underscore.js (http://underscorejs.org/) and jQuery (http://jquery.com/). All other libraries and technologies will be properly introduced as they are used throughout this book.

Note

Although this book focuses on Backbone.js applications, the test techniques and technologies we introduce should easily carry over to other frontend JavaScript frameworks and web applications. There are a lot of great frameworks in the frontend ecosystem besides Backbone.js—try one of them!

Designing an application and test repository structure

Setting up a test infrastructure first requires a plan as to where all the parts and pieces will go. We will start with a simple directory structure for a code repository as follows:

app/
  index.html
  css/
  js/
    app/
    lib/

test/
  test.html
  js/
    lib/
    spec/

The app/index.html file contains the web application, while test/test.html provides the test driver page. Application and test libraries are respectively contained in the app/js/ and test/js/ directories.

Note

This is just one way to organize a Backbone.js application and tests. Other directory layouts may be more appropriate, and you should feel free to follow your own conventions and preferences in light of the specific development project at hand.

The Backbone.js application and component files (models, views, routers, and so on) are placed in app/js/app/, which may look something like the following:

app/js/app/
  app.js
  models/
    model-a.js
    ...
  views/
    view-a.js
    ...
  ...

The core application libraries are stored in app/js/lib/, which should include the libraries needed to drive the actual application:

app/js/lib/
  backbone.js
  jquery.js
  underscore.js
  ...

The test libraries and suites get a separate directory, test/js/, which isolates the test code from the application to avoid inadvertently introducing application dependencies on test functions or libraries:

test/js/
  lib/
    mocha.js
    mocha.css
    chai.js
    sinon.js
  spec/
    first.spec.js
    second.spec.js
    ...

Now that we have an abstract application and a test layout, we need to fill in all the pieces and populate directories with libraries, web pages, and test files.

Getting the test libraries

The ecosystem of frontend JavaScript test frameworks is quite rich, with libraries supporting different paradigms, features, and functionality. Choosing tools from this collection is a difficult task, without clear correct answers. In this book, we have settled on three complementary libraries, Mocha, Chai, and Sinon.JS, that provide an aggregate set of features particularly well suited for testing Backbone.js applications. In addition to these libraries, we will use the PhantomJS headless web browser to automate our test infrastructure and run tests from the command line.

Note

Server-side JavaScript testing with Mocha, Chai, and Sinon.JS

Beyond the browser, JavaScript has seen a meteoric rise as a server technology via the immensely popular Node.js framework, supplanting traditional server-side languages and providing developers with a single-language web application stack. Although we will only discuss frontend testing in this book, the three core testing libraries we use are all available as server-side testing modules for Node.js. There are some non-trivial differences in integration and use (for example, Mocha reports are run from the command line and not a browser), but many of the general testing and application design concepts we will cover in this book equally apply to Node.js server applications, and you can conveniently use exactly the same test libraries in your frontend and backend development.

Following the repository structure discussed previously, we will download each of the test library files to the test/js/lib/ directory. After this, we will be ready to write and run a test web page against the libraries. Note that although we pick specific library versions in this book to correspond with the downloadable examples code, we generally recommend using the most recent versions of these libraries.

Mocha

The Mocha (http://visionmedia.github.io/mocha/) framework supports test suites, specs, and multiple test paradigms. Some of the nifty features offered by Mocha include frontend and backend integration, versatile timeouts, slow test identification, and many different test reporters.

To run Mocha tests in a browser, we just need two files—mocha.js and mocha.css. For version 1.9.0, both these files are available from GitHub at the following locations:

Note

At the time this book went to press, the most current versions of Mocha (1.10.0 and above) have introduced an incompatibility with the Mocha-PhantomJS automation tool that we will use later in this book. You can watch the Mocha (https://github.com/visionmedia/mocha/issues/770) and Mocha-PhantomJS (https://github.com/metaskills/mocha-phantomjs/issues/58) tickets for status updates and possible future fixes.

The JavaScript (mocha.js) file contains the library code and the CSS (mocha.css) file provides the styles for the HTML reporter page. With these files in place, we can organize our tests into suites and specs, run our tests, and get a usable report of test results.

Note

Why Mocha?

Mocha is just one framework from an overall collection of great test libraries. Some of the strengths of the Mocha framework include solid asynchronous test support, server-side compatibility, alternative test interfaces, and flexible configurability. But, we could just as easily go with another test library.

As an example of an alternate framework, Jasmine (http://pivotal.github.io/jasmine/) from Pivotal Labs is another enormously popular JavaScript testing framework. It provides test suite and spec support, a built-in assertion library, and many more features (including test spies)—it is essentially an all-in-one framework. By contrast, Mocha is quite flexible, but you have to add additional components. For example, we leverage Chai for assertions and Sinon.JS for mocks and stubs in the test infrastructure of this book.

Chai

Chai (http://chaijs.com/) is a test assertion library that offers an extensive API, support for Behavior-Driven Development (BDD) and Test-Driven Development (TDD) test styles, and a growing plugin ecosystem. BDD and TDD will be introduced in more detail in Chapter 2, Creating a Backbone.js Application Test Plan. In particular, we will use Chai's chainable test functions to write assertions that read very closely to natural language, allowing tests to maximize comprehensibility while minimizing the need for explanatory code comments.

For integration, we need to download a single library file—chai.js. The version (1.7.1) that we want is available at https://raw.github.com/chaijs/chai/1.7.1/chai.js.

Alternatively, the current stable version of Chai can be found at http://chaijs.com/chai.js.

Sinon.JS

The Sinon.JS library (http://sinonjs.org/) provides a powerful suite of test spies, stubs, and mocks. Spies are functions that analyze and store information about an underlying function and can be used to verify historical behavior of the function under test. Stubs are spies that can replace a function with a different behavior more amenable to testing. Mocks spy on and stub functions as well as verify that certain behavior has occurred during test execution. We will explain these tools in more detail throughout this book.

In practice, Backbone.js applications comprise many different and constantly interacting parts, making our goal of testing isolated program components difficult. A mocking library such as Sinon.JS will allow us to separate testable application behaviors and focus on one thing (for example, a single view or a model) at a time.

Like Chai, we just need a single JavaScript file to use Sinon.JS in our tests. Versioned releases—we will use version 1.7.3—are available at either of the following locations:

Installation of Sinon.JS, along with Mocha and Chai, completes the acquisition phase of our test infrastructure creation.

Setting up and writing our first tests

Now that we have the base test libraries, we can create a test driver web page that includes the application and test libraries, sets up and executes the tests, and displays a test report.

Tip

Downloading the example code

The source code for all snippets and code examples in this book is available online. Files and tests for each chapter can be found by number in the chapters directory. See the Preface for download locations and installation instructions.

The examples are best used as a helpful check on your own progress after a chapter has been finished and you have applied the lessons and exercises to your own code and applications. As a gentle admonition, we encourage you to resist the temptation to copy and paste code or files from the examples. The experience of writing and adapting the code on your own will allow you to better internalize and understand the testing concepts needed to become an adept Backbone.js tester.

The test driver page

A single web page is typically used to include the test and application code and drive all frontend tests. Accordingly, we can create a web page named test.html in the chapters/01/test directory of our repository starting with just a bit of HTML boilerplate—a title and meta attributes:

<html>
  <head>
    <title>Backbone.js Tests</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">

Then, we include the Mocha stylesheet for test reports and the Mocha, Chai, and Sinon.JS JavaScript libraries:

    <link rel="stylesheet" href="js/lib/mocha.css" />
    <script src="js/lib/mocha.js"></script>
    <script src="js/lib/chai.js"></script >
    <script src="js/lib/sinon.js"></script>

Next, we prepare Mocha and Chai. Chai is configured to globally export the expect assertion function. Mocha is set up to use the bdd test interface and start tests on the window.onload event:

    <script>
      // Setup.
      var expect = chai.expect;
      mocha.setup("bdd");

      // Run tests on window load event.
      window.onload = function () {
        mocha.run();
      };
    </script>

After the library configurations, we add in the test specs. Here we include a single test file (that we will create later) for the initial test run:

    <script src="js/spec/hello.spec.js"></script>
  </head>

Finally, we include a div element that Mocha uses to generate the full HTML test report. Note that a common alternative practice is to place all the script include statements before the close body tag instead of within the head tag:

  <body>
    <div id="mocha"></div>
  </body>
</html>

And with that, we are ready to create some tests. Now, you could even open chapters/01/test/test.html in a browser to see what the test report looks like with an empty test suite.

Adding some tests

While test design and implementation is discussed in far more detail in subsequent chapters, it is sufficient to say that test development generally entails writing JavaScript test files, each containing some organized collection of test functions. Let's start with a single test file to preview the testing technology stack and give us some tests to run.

The test file chapters/01/test/js/spec/hello.spec.js creates a simple function (hello()) to test and implements a nested set of suites introducing a few Chai and Sinon.JS features. The function under test is about as simple as you can get:

window.hello = function () {
  return "Hello World";
};

The hello function should be contained in its own library file (perhaps hello.js) for inclusion in applications and tests. The code samples simply include it in the spec file for convenience.

The test code uses nested Mocha describe statements to create a test suite hierarchy. The test in the Chai suite uses expect to illustrate a simple assertion. The Sinon.JS suite's single test shows a test spy in action:

describe("Trying out the test libraries", function () {
  describe("Chai", function () {
    it("should be equal using 'expect'", function () {
      expect(hello()).to.equal("Hello World");
    });
  });

  describe("Sinon.JS", function () {
    it("should report spy called", function () {
      var helloSpy = sinon.spy(window, 'hello');

      expect(helloSpy.called).to.be.false;
      hello();
      expect(helloSpy.called).to.be.true;
      hello.restore();
    });
  });
});

Not to worry if you do not fully understand the specifics of these tests and assertions at this point, as we will shortly cover everything in detail. The takeaway is that we now have a small collection of test suites with a set of specifications ready to be run.

Running and assessing test results

Now that all the necessary pieces are in place, it is time to run the tests and review the test report.

The first test report

Opening up the chapters/01/test/test.html file in any web browser will cause Mocha to run all of the included tests and produce a test report:

Test report

This report provides a useful summary of the test run. The top-right column shows that two tests passed, none failed, and the tests collectively took 0.01 seconds to run. The test suites declared in our describe statements are present as nested headings. Each test specification has a green checkmark next to the specification text, indicating that the test has passed.

Test report actions

The report page also provides tools for analyzing subsets of the entire test collection. Clicking on a suite heading such as Trying out the test libraries or Chai will re-run only the specifications under that heading.

Clicking on a specification text (for example, should be equal using 'expect') will show the JavaScript code of the test. A filter button designated by a right triangle is located to the right of the specification text (it is somewhat difficult to see). Clicking the button re-runs the single test specification.

The test specification code and filter

The previous figure illustrates a report in which the filter button has been clicked. The test specification text in the figure has also been clicked, showing the JavaScript specification code.

Tip

Advanced test suite and specification filtering

The report suite and specification filters rely on Mocha's grep feature, which is exposed as a URL parameter in the test web page. Assuming that the report web page URL ends with something such as chapters/01/test/test.html, we can manually add a grep filter parameter accompanied with the text to match suite or specification names.

For example, if we want to filter on the term spy, we would navigate a browser to a comparable URL containing chapters/01/test/test.html?grep=spy, causing Mocha to run only the should report spy called specification from the Sinon.JS suite. It is worth playing around with various grep values to get the hang of matching just the suites or specifications that you want.

Test timing and slow tests

All of our tests so far have succeeded and run quickly, but real-world development necessarily involves a certain amount of failures and inefficiencies on the road to creating robust web applications. To this end, the Mocha reporter helps identify slow tests and analyze failures.

Tip

Why is test speed important?

Slow tests can indicate inefficient or even incorrect application code, which should be fixed to speed up the overall web application. Further, if a large collection of tests run too slow, developers will have implicit incentives to skip tests in development, leading to costly defect discovery later down the deployment pipeline.

Accordingly, it is a good testing practice to routinely diagnose and speed up the execution time of the entire test collection. Slow application code may be left up to the developer to fix, but most slow tests can be readily fixed with a combination of tools such as stubs and mocks as well as better test planning and isolation.

Let's explore some timing variations in action by creating chapters/01/test/js/spec/timing.spec.js with the following code:

describe("Test timing", function () {
  it("should be a fast test", function (done) {
    expect("hi").to.equal("hi");
    done();
  });

  it("should be a medium test", function (done) {
    setTimeout(function () {
      expect("hi").to.equal("hi");
      done();
    }, 40);
  });

  it("should be a slow test", function (done) {
    setTimeout(function () {
      expect("hi").to.equal("hi");
      done();
    }, 100);
  });

  it("should be a timeout failure", function (done) {
    setTimeout(function () {
      expect("hi").to.equal("hi");
      done();
    }, 2001);
  });
});

We use the native JavaScript setTimeout() function to simulate slow tests. To make the tests run asynchronously, we use the done test function parameter, which delays test completion until done() is called. Asynchronous tests will be explored in more detail in Chapter 3, Test Assertions, Specs, and Suites.

The first test has no delay before the test assertion and done() callback, the second adds 40 milliseconds of latency, the third adds 100 milliseconds, and the final test adds 2001 milliseconds. These delays will expose different timing results under the Mocha default configuration that reports a slow test at 75 milliseconds, a medium test at one half the slow threshold, and a failure for tests taking longer than 2 seconds.

Next, include the file in your test driver page (chapters/01/test/test-timing.html in the example code):

    <script src="js/spec/timing.spec.js"></script>

Now, on running the driver page, we get the following report:

Test report timings and failures

This figure illustrates timing annotation boxes for our medium (orange) and slow (red) tests and a test failure/stack trace for the 2001-millisecond test. With these report features, we can easily identify the slow parts of our test infrastructure and use more advanced test techniques and application refactoring to execute the test collection efficiently and correctly.

Test failures

A test timeout is one type of test failure we can encounter in Mocha. Two other failures that merit a quick demonstration are assertion and exception failures. Let's try out both in a new file named chapters/01/test/js/spec/failure.spec.js:

// Configure Mocha to continue after first error to show
// both failure examples.
mocha.bail(false);

describe("Test failures", function () {
  it("should fail on assertion", function () {
    expect("hi").to.equal("goodbye");
  });

  it("should fail on unexpected exception", function () {
    throw new Error();
  });
});

The first test, should fail on assertion, is a Chai assertion failure, which Mocha neatly wraps up with the message expected 'hi' to equal 'goodbye'. The second test, should fail on unexpected exception, throws an unchecked exception that Mocha displays with a full stack trace.

Note

Stack traces on Chai assertion failures vary based on the browser. For example, in Chrome, no stack trace is displayed for the first assertion while one is shown in Safari. See the Chai documentation for configuration options that offer more control over stack traces.

Test failures

Mocha's failure reporting neatly illustrates what went wrong and where. Most importantly, Chai and Mocha report the most common case—a test assertion failure—in a very readable natural language format.

Summary

In this chapter, we introduced an application and test structure suitable for development, gathered the Mocha, Chai, and Sinon.JS libraries, and created some basic tests to get things started. Then, we reviewed some facets of the Mocha test reporter and watched various tests in action—passing, slow, timeouts, and failures.

In the next chapter, we will integrate a Backbone.js application as the target of our test framework and learn how to test, isolate, and verify program behavior throughout the course of application development.