Building Scalable Apps with Redis and Node.js

To create our own middleware, all we have to do is create a function that accepts the parameters req, res, and next. Inside of this function, we should have access to request and response, and the ability to tell Express to move on to the next piece of middleware.

To add a middleware layer to Express, we use app.use(), which allows us to take a middleware function. So, let's create our own 404 middleware function! Create a directory named middleware and a file called errorhandlers.js, before putting the following code in the file:

exports.notFound = function notFound(req, res, next){
  res.send(404, 'You seem lost. You must have taken a wrong turn back there.');
};

Now, update app.js and put this code right after the initial block of variable declarations, but before we define routes. This should be line 4. This is shown as follows:

var errorHandlers = require('./middleware/errorhandlers');
app.use(errorHandlers.notFound);

We didn't call the next function here because no other middleware matches this route at this point. We can safely send the response by letting the user know that they are lost. Let's fire this up and check it out. We will type a nonexistent route, http://localhost:3000/notfound, into our browser. So far, everything looks good. Now, let's try a known route, http://localhost:3000/. Uh-oh! Everything is responding as not found. What did we do wrong?

How do we fix this issue? Pretty easily, in fact. We will add in another piece of middleware before our notFound handler. It will be the app.router middleware, since this is the function that maps all the routes. If a request matches a defined route, then it will execute the function we have defined for that route and return a response. If it doesn't match anything, the next middleware will be called. You should then open up app.js and move app.use(errorHandlers.notFound) under the routes we have created:

app.get('/', routes.index);
app.get('/login', routes.login);
app.post('/login', routes.loginProcess);
app.get('/chat', routes.chat);
app.use(errorHandlers.notFound);

This will check the request to see if it matches a route. If not, run the next middleware, which is the notFound function. Our app is now running how we expect it to. Try loading all the routes, and then run the routes that we know will create a 404 error to test our new middleware.

Let's add a few more pieces of middleware. First off, let's add a logging middleware. Create log.js under the middleware folder and put the following code in:

exports.logger = function logger(req, res, next){
  console.log(req.url);
  next();
};

Then, modify app.js, and add this as the first middleware:

var errorHandlers = require('./middleware/errorhandlers');
var log = require('./middleware/log');
app.use(log.logger);

app.get('/', routes.index);
app.get('/login', routes.login);
app.post('/login', routes.loginProcess);
app.get('/chat', routes.chat);
app.use(errorHandlers.notFound);

Each request will log the URL to the console. We did not modify the request in any way, so it continues down the middleware path to the route or the notFound handler. We can change this to write to a file or a database, but we are keeping things simple for now (is this foreshadowing? It probably is!). Also, we do not need to modify the request or response in any way.

Next, we will add the ability to serve static assets. Most sites use CSS and JavaScript, and we don't want to send these files through the view rendering engine (a concept we will get to later in this chapter). Express comes with middleware that can serve static files. So, create a folder in our project called static, and then create a file called static.txt, putting whatever you want in the file. Now, add the static middleware right above the router, as follows:

app.use(log.logger);
app.use(express.static(__dirname + '/static'));

Anything you put in this folder will be served. Browse to http://localhost:3000/static.txt, and you should see whatever you added to the file.

Finally, let's add an error handler. This middleware has a different function signature. It takes the four parameter functions of err, req, res, and next. This conforms to the node practice of passing the error as the first parameter. We will add the error handler inside middleware/errorhandlers.js. To conclude, add the following code to the file:

exports.error = function error(err, req, res, next){
  console.log(err);
  res.send(500, 'Something broke. What did you do?');
};

Here is our final middleware stack in app.js:

app.use(log.logger);
app.use(express.static(__dirname + '/static'));
app.get('/', routes.index);
app.get('/login', routes.login);
app.post('/login', routes.loginProcess);
app.get('/chat', routes.chat);

app.use(errorHandlers.error);
app.use(errorHandlers.notFound);

At this point, we cannot test the error handler. Every request we create does not throw an error, so let's create a route that actually does. Add this to the end of our route definitions:

app.get('/error', function(req, res, next){
  next(new Error('A contrived error'));
});

Remember that a route is just another piece of middleware. We create an error here to pass to the error handler. The only middleware that will match the function signature with an error is our new error handler. If we go to /error, we will see that our logging middleware writes to the console, followed by the error middleware writing our error to the console. It then concludes with Something broke. What did you do?. Our little website is now not blowing up on errors and logging everything.

Right now, we are serving HTTP responses based on routes and wired-up logging, 404 not found error page, and error handling middleware. This is all in roughly 20 lines of code in app.js. Not too bad!

Express 3 lets each template-rendering library implement its own layout, which means it does not force any set way on the layout. This is different from Express 2 as it's defaulted to using a file named layout in the views directory. You will have to be careful that you do not get information about Express 2 when searching on the Internet. Our package of EJS does not have layout support, but we have already installed a package that does, which is express-partials.

We have to do a little setup to let Express know that we are using partials. The first step is to get a reference to our new library, which you can do by:

//all the other variables declarations at the top of app.js
var partials = require('express-partials');

Next, we add it as middleware. The only caveat here is that it has to be before our routes. This makes sense because we will have to render our complete page before the server sends a response.

app.use(partials());
//all other middleware functions

Finally, we can add a view option for the default layout.

//after variable declarations 
app.set('view options', {defaultLayout: 'layout'});
//but before the middleware

Setting the default layout is not required, but I will recommend it. Otherwise, we will have to define the layout for every view. We can override the default layout (or set it in the first place) using the render function:

res.render('index', {layout: 'layout', title: 'Index'});

Now that we have our layout setup, we can create it and update all our routes to use the layout. Create a file called layout.ejs in the views folder. Add the following code to it:

<!DOCTYPE html>
<html>
<head>
  <title><%= title %></title>
</head>
<body>
<%- body %>
</body>
</html>

Now, we can create two more files called login.ejs and chat.ejs. Finally, we have to update each res.render function to have the name of the file to render and a title.

exports.login = function chat(req, res){
  res.render('login', {title: 'Login'});
};
exports.chat = function chat(req, res){
  res.render('chat', {title: 'Chat'});
};

Now, all the routes will return valid HTML.

Let's go ahead and make our layout pretty. We will add the HTML code that our app will use, and the CSS framework we will be using is Bootstrap. It's great to get projects off the ground as we can start to be concerned with the code as opposed to the design. We will use a theme named Cosmo from http://bootswatch.com/ so that our site does not look like a default Bootstrap site. We can get the CSS framework from http://bootswatch.com/cosmo/. Once you have downloaded the CSS file, create a new directory named css under the static directory. The path from root should be static/css/cosmo.min.css.

After carrying out all of these tasks, here is what our layout should look like:

<!DOCTYPE html>
<html>
<head>
    <title><%= title %></title>
    <link rel="stylesheet" href="css/cosmo.min.css">
</head>
<body>
<div class="container">
  <div class="row">
      <div class="col-sm-4"><h1>PacktChat</h1></div>
  </div>
  <div class="row">
  <%- body %>
  </div>
</div>
</body>
</html>

Refer to the following screenshot to see the index page in the new layout:

The latest version of Express has taken out much of the middleware that was previously included in Express. This is important when migrating from Express 3 to 4. We will use the cookie-parser package, which should already be installed. We will now add cookie parsing to our app. It is a middleware, so we will put it with all the other middleware. Remember that the middleware is processed in order, so that we can add each before the middleware that will actually use it, which is our routes. We will do this by adding a variable declaration and another function in the middleware stack.

//with all the other requires at the top of the file
var cookieParser = require('cookie-parser');
//in the middleware stack
app.use(express.static(__dirname + '/static'));
app.use(cookieParser());

We will repeat this pattern many times over this chapter and the next. Anytime there is require, it will be at the top of the file. The code that comes along with the book will also declare all the variables together. There will be one var, and each line will have require with a comma instead of a semicolon. Whether there is one var declaration or many, the code will still run. Further down in the file, we will use our new variable. I will try to include landmarks in the code to help, but we will add a lot of code in many places at times. Refer to the code that is supplied with the book.

The cookie parser gives us access to req.cookies. This will be an object that we can read the values out of. At this point, if we run our application, nothing will be different. We have not set cookies, nor are we trying to access them. Let's change that.

First, let's set up our views to show us the cookies in the request. In index.ejs under views, let's add the section. The file should look similar to the following code:

Index
<div>Cookie passed: <%= cookie %></div>

We now have to pass the cookie to our view. You should edit routes/index.js and add this to our view function. Note that we do not need to specify a layout because we have a default layout set with app.set('view options', {defaultLayout: 'layout'}).

exports.index = function index(req, res){
  res.render('index', {title: 'Index', cookie: JSON.stringify(req.cookies)});
};

Let's check it out in the browser. We should see that we have a blank cookie object in our request. It is time to create our own cookie. Open the JavaScript console in the browser (I am using Google Chrome) and type this in:

document.cookie="test=Test Cookie"

Refresh the page and see that it has our cookie in it. We can see that the request cookie object is just a simple JavaScript object, as seen in the following screenshot:

Next, we will set a cookie from the server. Express has a simple way to do this: in our index.js file under routes, let's add a cookie in the index function:

exports.index = function index(req, res){
  res.cookie('IndexCookie', 'This was set from Index');
  
res.render('index', {title: 'Index', cookie: JSON.stringify(req.cookies)});
};

Restart the node and load the page twice. The first restart will set the cookie, and the second restart will read it into our response. From the following screenshot, you should now see both cookies on our page:

You can also easily get rid of cookies by using clearCookie off the response object:

res.clearCookie('IndexCookie');

If you want to do it from the browser side, you can usually get a list of current cookies. In Chrome, this is in the developer tools. Click on the menu button in the upper right and navigate to Tools | Developer Tools. Then click on Resources | Cookies. You can then right-click on a specific cookie in the list and delete the cookie or select Clear All to delete all the cookies, as shown in the following screenshot:

By now, you should be feeling good about adding and removing cookies to requests and responses, so now let's see how to tie these cookies to a session.

Sessions allow us to store data about requests that are tied together with a cookie. HTTP is stateless, but cookies that map back to a session allow us to know that this is the same browser making multiple requests. You should be able to guess by now that Express comes with a great session middleware.

The first thing to know is that we need to store our sessions somewhere. For now, we will use a memory store.

You should add this to our variable declarations at the top of app.js:

var session = require('express-session');

Next, add the middleware. You should remember to add it under our cookieParser middleware, as follows:

app.use(cookieParser());
app.use(session());

The express session uses cookies, so the cookie object needs to be present before it can use the session.

Now, we can use our session. We will update our index page to show what is stored in our session. Edit index.ejs under views to display a session:

Index
<div>Cookie passed: <%= cookie %></div>
<div>Session: <%= session %></div>

The session middleware adds a new object to our request, which is req.session. Let's pass this to the view from index.js under middleware:

function index(req, res){
  res.cookie('IndexCookie', 'This was set from Index');
  res.render('index', {title: 'Index', cookie: JSON.stringify(req.cookies), session: JSON.stringify(req.session)});
};

Once you load this up, you will find that we get an error. If we check our console, as you can see from the following screenshot, we need to add a secret option for sessions:

We can now do this by revisiting our session middleware and adding a secret option:

app.use(session({secret: 'secret'}));

The secret option uses the string we pass in to create a hash of our session ID, so we can tell if someone has tried to tamper with our cookie (also known as a request forgery). We just covered ways by which users can easily delete and create any cookie that they want. If our cookie had a session ID in it, which for example could be 1234, a user could delete that cookie and create a new one with a session ID of 1235. As far as the server knows, the next request comes from the user who has session 1235. A hashed session ID makes this much more difficult. If the user does not know the secret (don't actually use secret or 123456, use something such as http://randomkeygen.com/ or http://www.guidgenerator.com/ to get a unique secure secret), then their ability to create a valid token is reduced. This is a very contrived example, but it should illustrate why we need this.

Reload the node and refresh twice. We can now see our session and the cookie that was created in the following screenshot:

We can test our security by deleting our connect.sid cookie and creating a new one. On the next request, we will get a new connect.sid cookie set.

Let's build a simple page counter in the session. On each request, we will increment a counter. We can do this easily by adding a middleware function. We only need to remember to add it under the session middleware so that we have access to req.session; we will write this function inline as we are not going to keep it in our final middleware stack. Add this to the stack right under session:

app.use(function(req, res, next){
  if(req.session.pageCount)
    req.session.pageCount++;
  else
    req.session.pageCount = 1;
  next();
});

Test it by going around and loading a bunch of pages. The pageCount session variable should track each different request you make. The request could be a 404 or even an error. Our middleware runs and adds to the total before any error handling. One thing to remember is that only our index view will output pageCount. After testing this, we can remove the middleware.

One limitation to how we have set this up is that only the node instance that created the session also has access to it. If you run multiple node instances, you will need to have a different session store from memory.

Redis is an in-memory key-value store. We will use Redis to hold the session ID as a key and the session data as a value. It is important to note that we will not get into what Redis is and how to install it here as Chapter 5, Adopting Redis for Application Data, will cover the topic. Also, we will not cover the security issues with Redis now as we just want to get it working for our sessions. However, we will cover how to add it as an Express session store.

We will use the two packages redis and connect-redis. To use a Redis store, we assume that we are running Redis locally and that Redis' version is above 2.0.0 (the latest version, as of writing this book, is 2.8.6, so this isn't a huge hurdle). First, let's change our reference to the memory store so that our variable session will point to a connect-redis instance. Change these variable declarations in app.js:

var session = require('express-session');
var RedisStore = require('connect-redis')(session);

Connect-redis extends the session. We can now set up our middleware. Change our session middleware to this:

app.use(session({
  secret: 'secret',
  saveUninitialized: true,
  resave: true,
  store: new RedisStore(
    {url: 'redis://localhost'})
  })
);

We use the same secret, but we will now create a new RedisStore object with an options object using the Redis server's URL. This URL can take a username, password, and port, if all of these were not the default values. At this point, we can restart our server and load up our index page. It should be working in exactly the same way as it was with an in-memory store. We also have a couple of other options. If these are not added, a warning is thrown.

Let's actually take a peek into what is happening here. We know at this point that our session is tracked with a cookie, but unfortunately, this is a signed value. We can get access to this by changing our cookieParser middleware to use the same secret as the session middleware. The following line of code is what our new cookieParser line should look like:

app.use(cookieParser('secret'));

Remember that the secret passed must match the one used for the session. This is because the session middleware creates the cookie and the cookieParser middleware reads it out. We will now have req.signedCookies. Any signed cookie will be here, so it is time to test this out. We will need to update index.ejs in the View folder and index.js in the routes folder provided in the code bundle.

The index.ejs file in the views folder looks like:

Index
<div>Cookie passed: <%= cookie %></div>
<div>Signed Cookie passed: <%= signedCookie %></div>
<div>Session: <%= session %></div>

The index.js file in the routes folder looks like:

exports.index = function index(req, res){
  res.cookie('IndexCookie', 'This was set from Index');
  res.render('index', {title: 'Index', 
    cookie: JSON.stringify(req.cookies), 
    session: JSON.stringify(req.session), 
    signedCookie: JSON.stringify(req.signedCookies)});
};

From the following screenshot, you can see that our unsigned cookies will be first and our connect.sid cookie will be second:

The browser will still get the signed cookie, as you can see in the following screenshot:

Without getting too much into Redis, we will look up our session in Redis. We can quickly install Redis on Mac OS X by running the following command:

brew install redis

We can then launch redis-cli (which we should now have if we have Redis installed. If you face issues, jump to Chapter 5, Adopting Redis for Application Data). We can now run a command against Redis. connect-redis will prepend sess: to all the session keys in Redis. To see our session, we will run GET sess:YOUR-SESSION-ID, as shown in the following command line:

$ redis-cli
127.0.0.1:6379> GET sess:0DMsXhobExvbCL3FFeYqRGWE
"{\"cookie\":{\"originalMaxAge\":null,\"expires\":null,\"httpOnly\":true,\"path\":\"/\"}}"

We can see that this returns our session object as an escaped string. You can compare this to the object that was returned from our response and see that it is the same. We have successfully moved our sessions to a data store that can be accessed from multiple servers. One of the most basic ideas of creating a scalable application is not to keep any shared state on the local instance. Previously, with the memory store for sessions, each server had its own state. Now, we can have multiple servers share the state. Here, we are using Redis, but you can use any data store to do this (which is not limited to memcache, MongoDB, Postgres, and many others). We are not going to do this here in this chapter, but we have started to prepare our app to be scalable. Another thing to note is that the Redis server is running on localhost. For a production-ready scalable application, Redis will be moved to a separate server or even several servers.

Let's clean up our views a little. You definitely do not want to send all of a user's session data to them. In index.ejs in the views folder, remove everything except for Index. In index.js in the routes folder, drop all the other attributes except for title, and remove the line that sets the cookie. This is shown as follows:

exports.index = function index(req, res){
  res.render('index', {title: 'Index'});
};

We are missing one huge security piece—the Cross-Site Request Forgery (CSRF) protection. Anyone can create a form and then tell it to process against our site. As far as Express is concerned, it is just another request, which is where CSRF comes in. It is essentially a token that is put into the form and a store on the server in the session. When the form is submitted, the server checks the token. If they do not match, the server knows the form is not from us and rejects it. It is time to implement it!

First, we add a..., you've got it, a middleware! Add the following code after your session and body parsing middleware:

//variable declarations
Var csrf = require('csurf');
//middleware stack
app.use(bodyParser.json());
app.use(bodyParser.urlencoded());
app.use(csrf());

By doing this, it will add the CSRF token to our session.

With CSRF loaded, let's try to submit our form using the POST method. As you can see in the following screenshot, we get the error Something broke. What did you do?

The CSRF middleware threw a 403 status code forbidden error. Our CSRF protection is working exactly as planned as we did not add the token to the form. This is exactly what will happen if someone with nefarious purposes tried to submit a form against our site with the token.

To fix this, let's add another piece of middleware (note that our app is 90 percent middleware and only 10 percent views). We will create a utility function file by creating a utilities.js file under the middleware directory. Inside this file, add the following:

module.exports.csrf = function csrf(req, res, next){
  res.locals.token = req.csrfToken();
  next();
};

Res.locals is a special object that is available to all templates that have access to this response. It will make all of its attributes available for the template to use. This is great for middleware as the only other way that we covered to get variables to the template was by passing them in through render(). Create a new variable at the top of app.js:

Var util = require('./middleware/utilities');

Add it as middleware directly under our CSRF middleware:

app.use(csrf());
app.use(util.csrf);

It seems a little redundant to add two straight CSRF middlewares, but they serve different functions. The first is the built-in middleware that will add the token to the session. The other is our utility that will take it from the session and make it available to the template. Maintainable middleware should have one simple single focus.

We can now edit our form to send this token, so right under the form declaration, add the following input tag:

<input type="hidden" name="_csrf" value="<%= token %>">

Our forms should now process and our middle stack should look similar to the following code:

app.use(partials());
app.use(log.logger);
app.use(express.static(__dirname + '/static'));
app.use(cookieParser('secret'));
app.use(session({
  secret: 'secret',
  saveUninitialized: true,
  resave: true,
  store: new RedisStore(
    {url: 'redis://localhost'})
  })
);app.use(bodyParser.json());
app.use(bodyParser.urlencoded());
app.use(csrf());
app.use(util.csrf);
app.get('/', routes.index);
app.get('/login', routes.login);
app.post('/login', routes.loginProcess);
app.get('/chat', routes.chat);
app.use(errorHandlers.error);
app.use(errorHandlers.notFound);

We have now learned all the basic building blocks of using Express. We will now take them and see how to build a functioning site.

A route map allows us to use a programmable name for a specific URL. Our new config does this for login and logout. A quick check of our application shows that we are using the string '/login' in four different places. Let's all make it in one place.

We will do this by creating another piece of middleware. Any backend file can get access to this by loading the config module with require(), but the templates do not have this ability. Also, if they did, we do not want to put a boilerplate requirement at the top of every view template. This is where our new middleware comes in. Add a reference at the top of the utitlities.js file present in the middleware folder, and create a new function in the file.

var config = require('../config');
//the other functions or you could put this at the top
exports.templateRoutes = function templateRoutes(req, res, next){
  res.locals.routes = config.routes;

  next();
};

We can see that this just adds the routes object from config to res.locals. Every template will now be able to use the login and logout routes.

Now, add it to the middleware stack. Thinking about our middleware execution order, we know that it just has to go before any templates render. The first middleware that renders is our app.router, so it must go before this.

app.use(flash());
app.use(util.templateRoutes);

Our app now has a config, so we must find all the references to the various settings.

The first thing to look at is app.js. We will need to add this object to the scope, so add this line as the last variable declaration:

Var config = require('./config');

Now, we must find each reference to the settings in our config. Let's update the secret.

app.use(cookieParser(config.secret));
app.use(session({
  secret: config.secret,
  saveUninitialized: true,
  resave: true,
  store: new RedisStore(
   {url: config.redisUrl})
  })
  );

We fixed the issue that we had initially posed. The only change that we now need to make is in config.js, and all the secrets will be set.

We can now set up the route map's routes. We will change three of our routes in app.js.

app.get(config.routes.login, routes.login);
app.post(config.routes.login, routes.loginProcess);
app.get(config.routes.logout, routes.logOut);

The app now uses the route map to determine the actual URL to bind to. However, now we have an issue: if we update the route in the config, we have some functions that have hard coded '/login'. This will throw a 404 error. We need to track down all the references to login and logout.

The first reference, utilites.js, present in the middleware folder, is the requireAuthentication function. We will update isAuthenticated to redirect back to config.routes.login.

module.exports.requireAuthentication = function requireAuthentication(req, res, next){
  if (req.session.isAuthenticated) {
    next();
  }else {
    res.redirect(config.routes.login);
  }
};

Next is our index.js present in the routes folder; it is the loginProcess function. We will need to redirect back to login on an auth failure.

//add the config reference
var config = require('../config');
//change the function
function loginProcess(req, res){
  var isAuth = util.auth(req.body.username, req.body.password, req.session);
  if (isAuth) {
    res.redirect('/chat');
  }else {
    req.flash('error', 'Wrong Username or Password');
    res.redirect(config.routes.login);
  }
};

The last two files that we will update are the partials. Each partial file has a hard-coded URL that we will change to use config.route as follows:

Here is where our middleware is valuable. Every template will have a route's object straight from the config. We are only using it for login and logout, but a site-wide URL should be here so that you can easily update it. For example, we can change the config.routes login and logout to '/account/login' and '/account/logout', and the app will not break.

Finally, we will update the port that the app listens on. This is in app.js:

app.listen(config.port);