Advanced Express Web Application Development

We will now begin the process of building a Vision application. We will start from scratch with a test-first approach. Along the way, we will explore some best practices and offer tips for when developing web applications with node and Express.

The Vision application will include the following features:

Feature: Heartbeat
As an administrator
I want to visit an endpoint
So that I can confirm the server is responding

Feature: List projects
As a vision user
I want to see a list of projects
So that I can select a project I want to monitor

Feature: Create project
As a vision user
I want to create a new project
So that I can monitor the activity of multiple repositories

Feature: Get a project
As a vision user
I want to get a project
So that I can monitor the activity of selected repositories

Feature: Edit a project
As a vision user
I want to update a project
So that I can change the repositories I monitor

Feature: Delete a project
As a vision user
I want to delete a project
So that I can remove projects no longer in use

Feature: List repositories
As a vision user
I want to see a list of all repositories for a GitHub account
So that I can select and monitor repositories for my project

Feature: List issues
As a vision user
I want to see a list of multiple repository issues in real time
So that I can review and fix issues

Feature: List commits
As a vision user
I want to see a list of multiple repository commits in real time
So that I can review those commits

Feature: Master Page
As a vision user  
I want the vision application served as a single page
So that I can spend less time waiting for page loads

Feature: Authentication
As a vision user
I want to be able to authenticate via Github
So that I can view project activity

The following screenshot is of our Vision application; it contains a list of projects, repositories, commits, and issues. The upper-right corner has a login link that we will use for authentication:

If you do not have node installed, visit: http://nodejs.org/download/.

There is also an installation guide on the node GitHub repository wiki if you prefer not to or cannot use an installer: https://github.com/joyent/node/wiki/Installation.

Let's install Express globally:

npm install -g express

If you have downloaded the source code, install its dependencies by running this command:

npm install

Let's start by creating a root project folder called vision and add a package.json file to it: ./package.json:

{
  "name": "chapter-1",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "start": "node app.js"
  }
  "dependencies": {
    "express": "3.x"
  }
}

Now that we have Express installed and our package.json file in place, we can begin to drive out our application with a test-first approach. We will now install two modules to assist us: mocha and supertest.

Mocha is a testing framework for node; it's flexible, has good async support, and allows you to run tests in both a TDD and BDD style. It can also be used on both the client and server side. Let's install Mocha with the following command:

npm install -g mocha –-save-dev

SuperTest is an integration testing framework that will allow us to easily write tests against a RESTful HTTP server. Let's install SuperTest:

npm install supertest –-save-dev

As an administrator
I want to visit an endpoint
So that I can confirm the server is responding

Let's add a test to ./test/heartbeat.js for our Heartbeat feature. This resource will get a status from the route /heartbeat and return a 200 Ok status code. Let's write our first integration test using Mocha and SuperTest. First off, create a folder named /test inside your vision folder.

Our test describes heartbeat; it expects the response to have a JSON content type and a status code equal to 200 Ok.

var app = require('../app')
, request = require('supertest');

describe('vision heartbeat api', function(){
  describe('when requesting resource /heartbeat', function(){
    it('should respond with 200', function(done){
      request(app)
      .get('/heartbeat')
      .expect('Content-Type', /json/)
      .expect(200, done);
    });
  });
});

Let's implement the Heartbeat feature; we start by creating a simple Express server, ./lib/express/index.js. We include the express and http modules and create an Express application. We then add an application setting via app.set called port and set it to 3000. We define a /heartbeat route via app.get with which we pass a request handler, function, that takes two parameters: req (request) and res (response). We use the response object to return a JSON response. We create an HTTP server with http.createServer by passing our Express application to it; we listen on port 3000 as defined in our application setting called port. We then export the application with module.exports; exporting the application allows us to test it.

var express = require('express')
  , http = require('http')
  , app = express();

app.set('port', 3000);

app.get('/heartbeat', function(req, res){
  res.json(200, 'OK')
});

http.createServer(app).listen(app.get('port'));
module.exports = app;

We now create ./app.js in the root of our project and export the express module:

module.exports = require('./lib/express');

To run our test, execute the following command:

mocha

You should then receive the response:

1 tests complete (14 ms)

If successful, try running the application by executing this command:

npm start

With the app running, run the following curl command in a new terminal and you can see our heartbeat JSON response return a 200 Ok status code:

curl -i http://127.0.0.1:3000/heartbeat

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 4
Date: Fri, 14 Jun 2013 08:28:50 GMT
Connection: keep-alive

One of the great things about working with a dynamic language and one of the things that has drawn me to node is the ability to easily do Test-Driven Development and continuous testing. Simply run Mocha with the -w watch switch and Mocha will respond when changes to our codebase are made, and will automatically rerun the tests:

mocha -w

Mocha is able to generate a code coverage report with a little help from JSCoverage . Install JSCoverage for your environment from http://siliconforks.com/jscoverage/. JSCoverage will parse source code and generate an instrumented version; this enables mocha to execute this generated code and create a report. We will need to update ./app.js.

module.exports = (process.env['NODE_ENV'] === "COVERAGE")
 ? require('./lib-cov/express')
 : require('./lib/express');

JSCoverage takes as arguments an input directory, and an output directory:

jscoverage lib lib-cov

Depending on your version of JSCoverage, you may need to add the –no-highlight switch:

jscoverage lib lib-cov --no-highlight

The following command will generate the coverage report, as shown in the following screenshot:

NODE_ENV=COVERAGE mocha -R html-cov > coverage.html

Nconf is a configuration tool that we will use to create hierarchical/environment configuration files for our application. Let's install Nconf:

npm install nconf --save

The first thing we will do is to move the following hardcoded port number from our Express application into our configuration:

app.set('port', 3000);

Let's create the module ./lib/configuration/index.js, which will allow us to to read configuration data from JSON files. We import the nconf module and define a constructor function, Config. We then load a configuration file based on the current environment and load the default configuration that holds non-environmental configuration data. We also define a function get(key), which accepts a key and returns a value. We will use this function to read configuration data:

var nconf = require('nconf');

function Config(){
  nconf.argv().env("_");
  var environment = nconf.get("NODE:ENV") || "development";
  nconf.file(environment, "config/" + environment + ".json");
  nconf.file("default", "config/default.json");
}

Config.prototype.get = function(key) {
  return nconf.get(key);
};

module.exports = new Config();

Let's write some configuration for our application. Add the following default configuration to ./config/default.json; this will be shared amongst all environments:

{
  "application": {
    "name": "vision"
  }
}

Now add the following configuration to the development, test, and coverage config files: ./config/development.json, ./config/test.json, and ./config/coverage.json.

{
  "express": {
    "port": 3000
  }
}

Let's change our Express server ./lib/express/index.js so that it reads express:port from configuration:

var express = require('express')
  , http = require('http')
  , config = require('../configuration')
  , app = express();

app.set('port', config.get("express:port"));

app.get('/hearbeat', function(req, res){
  res.json(200, 'OK');
});

http.createServer(app).listen(app.get('port'));

module.exports = app;

Express supports multiple options for application structure. Extracting elements of an Express application into separate files is one option; a good candidate for this is routes.

Let's extract our route heartbeat into ./lib/routes/heartbeat.js; the following listing simply exports the route as a function called index:

exports.index = function(req, res){
  res.json(200, 'OK');
};

Let's make a change to our Express server and remove the anonymous function we pass to app.get for our route and replace it with a call to the function in the following listing. We import the route heartbeat and pass in a callback function, heartbeat.index:

var express = require('express')
  , http = require('http')
  , config = require('../configuration')
  , heartbeat = require('../routes/heartbeat')
  , app = express();

app.set('port', config.get('express:port'));
app.get('/heartbeat', heartbeat.index);

http.createServer(app).listen(app.get('port'));
module.exports = app;

In order to handle a 404 Not Found response, let's add a 404 not found middleware. Let's write a test, ./test/heartbeat.js; the content type returned should be JSON and the status code expected should be 404 Not Found:

describe('vision heartbeat api', function(){
  describe('when requesting resource /missing', function(){
    it('should respond with 404', function(done){
      request(app)
      .get('/missing')
      .expect('Content-Type', /json/)
      .expect(404, done);
    })
  });
});

Now, add the following middleware to ./lib/middleware/notFound.js. Here we export a function called index and call res.json, which returns a 404 status code and the message Not Found. The next parameter is not called as our 404 middleware ends the request by returning a response. Calling next would call the next middleware in our Express stack; we do not have any more middleware due to this, it's customary to add error middleware and 404 middleware as the last middleware in your server:

exports.index = function(req, res, next){
    res.json(404, 'Not Found.');
};

Now add the 404 not found middleware to ./lib/express/index.js:

var express = require('express')
  , http = require('http')
  , config = require('../configuration')
  , heartbeat = require('../routes/heartbeat')
  , notFound = require('../middleware/notFound')
  , app = express();

app.set('port', config.get('express:port'));
app.get('/heartbeat', heartbeat.index);
app.use(notFound.index);

http.createServer(app).listen(app.get('port'));
module.exports = app;

Express comes with a logger middleware via Connect; it's very useful for debugging an Express application. Let's add it to our Express server ./lib/express/index.js:

var express = require('express')
  , http = require('http')
  , config = require('../configuration')
  , heartbeat = require('../routes/heartbeat')
  , notFound = require('../middleware/notFound')
  , app = express();

app.set('port', config.get('express:port'));
app.use(express.logger({ immediate: true, format: 'dev' }));
app.get('/heartbeat', heartbeat.index);
app.use(notFound.index);

http.createServer(app).listen(app.get('port'));
module.exports = app;

The immediate option will write a log line on request instead of on response. The dev option provides concise output colored by the response status. The logger middleware is placed high in the Express stack in order to log all requests.

We will now add logging to our application using Winston; let's install Winston:

npm install winston --save

The 404 middleware will need to log 404 not found, so let's create a simple logger module, ./lib/logger/index.js; the details of our logger will be configured with Nconf. We import Winston and the configuration modules. We define our Logger function, which constructs and returns a file logger—winston.transports.File—that we configure using values from our config. We default the loggers maximum size to 1 MB, with a maximum of three rotating files. We instantiate the Logger function, returning it as a singleton.

var winston = require('winston')
 , config = require('../configuration');

function Logger(){
  return winston.add(winston.transports.File, {
    filename: config.get('logger:filename'),
    maxsize: 1048576,
    maxFiles: 3,
    level: config.get('logger:level')
  });
}

module.exports = new Logger();

Let's add the Logger configuration details to our config files ./config/development.json and ./config/test.json:

{
  "express": {
    "port": 3000
  },
  "logger" : {
    "filename": "logs/run.log",
    "level": "silly",
  }
}

Let's alter the ./lib/middleware/notFound.js middleware to log errors. We import our logger and log an error message via logger when a 404 Not Found response is thrown:

var logger = require("../logger");

exports.index = function(req, res, next){
  logger.error('Not Found');
  res.json(404, 'Not Found');
};

Grunt is a task runner and a great way to automate your node projects. Let's add a simple grunt script to our project in order to automate running tests and code coverage. Let's install Grunt and Grunt CLI:

npm install -g grunt-cli
npm install grunt –-save-dev

The grunt-cafe-mocha is a grunt module for running mocha; this module will also allow us to automate code coverage reports:

npm install grunt-cafe-mocha –-save-dev

The grunt-jscoverage simply generates an instrumented version of our source code and writes it to ./lib-cov:

npm install grunt-jscoverage –-save-dev


The grunt-env allows you to set the current node environment, NODE_ENV:

npm install grunt-env  –-save-dev

Let's create a grunt file ./gruntfile.js. We load the grunt modules we just installed, and grunt.initConfig contains a configuration for each grunt module:

module.exports = function(grunt) {
  grunt.loadNpmTasks('grunt-jscoverage');
  grunt.loadNpmTasks('grunt-cafe-mocha');
  grunt.loadNpmTasks('grunt-env');

  grunt.initConfig({
    env: {
      test: { NODE_ENV: 'TEST' },
      coverage: { NODE_ENV: 'COVERAGE' }
    },
    cafemocha: {
      test: {
        src: 'test/*.js',
        options: {
          ui: 'bdd',
          reporter: 'spec',
        },
    },
    coverage: {
      src: 'test/*.js',
      options: {
        ui: 'bdd',
        reporter: 'html-cov',
        coverage: {
          output: 'coverage.html'
        }
      }
    },
  },
  jscoverage: {
    options: {
      inputDirectory: 'lib',
      outputDirectory: 'lib-cov',
      highlight: false
    }
  }
  });
  grunt.registerTask('test', [ 'env:test', 'cafemocha:test' ]);
  grunt.registerTask('coverage', [ 'env:coverage', 'jscoverage', 'cafemocha:coverage' ]);
};

The configuration for cafemocha contains two sections; one for running our tests and one for generating a code coverage report. In order to run our tests from grunt, execute the following command:

grunt test  

The following line registers a task that sets the environment using env and runs both the jscoverage and cafemocha:coverage tasks in sequence:

grunt.registerTask('coverage', [ 'env:coverage', 'jscoverage', 'cafemocha:coverage' ]);

In order to run our coverage from grunt, execute the following command:

grunt coverage

This command will generate the coverage report as described earlier.

We have put in place a fairly solid framework for developing our Vision project; we have implemented a simple feature, heartbeat, which when visited, simply informs us whether our Express server is up and running. We have automated various development tasks, such as running tests and creating code coverage reports. We also have in place some logging using Winston. In the next chapter, we will implement a web API.