Home Web Development Node.js Design Patterns

Node.js Design Patterns

By Mario Casciaro
books-svg-icon Book
eBook $32.99 $22.99
Print $54.99
Subscription $15.99 $10 p/m for three months
$10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
BUY NOW $10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
eBook $32.99 $22.99
Print $54.99
Subscription $15.99 $10 p/m for three months
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
About this book
Node.js is a massively popular software platform that lets you use JavaScript to easily create scalable server-side applications. It allows you to create efficient code, enabling a more sustainable way of writing software made of only one language across the full stack, along with extreme levels of reusability, pragmatism, simplicity, and collaboration. Node.js is revolutionizing the web and the way people and companies create their software. In this book, we will take you on a journey across various ideas and components, and the challenges you would commonly encounter while designing and developing software using the Node.js platform. You will also discover the "Node.js way" of dealing with design and coding decisions. The book kicks off by exploring the fundamental principles and components that define the platform. It then shows you how to master asynchronous programming and how to design elegant and reusable components using well-known patterns and techniques. The book rounds off by teaching you the various approaches to scale, distribute, and integrate your Node.js application.
Publication date:
December 2014
Publisher
Packt
Pages
454
ISBN
9781783287314

 

Chapter 1. Node.js Design Fundamentals

Some principles and design patterns literally define the Node.js platform and its ecosystem; the most peculiar ones are probably its asynchronous nature and its programming style that makes heavy use of callbacks. However, there are other fundamental components that characterize the platform; for example, its module system, which allows multiple versions of the same dependency to coexist in an application, and the observer pattern, implemented by the EventEmitter class, which perfectly complements callbacks when dealing with asynchronous code. It's therefore important that we first dive into these fundamental principles and patterns, not only for writing correct code, but also to be able to take effective design decisions when it comes to solving bigger and more complex problems.

Another aspect that characterizes Node.js is its philosophy. Approaching Node.js is in fact way more than simply learning a new technology; it's also embracing a culture and a community. We will see how this greatly influences the way we design our applications and components, and the way they interact with those created by the community.

In this chapter, we will learn the following topics:

  • The Node.js philosophy, the "Node way"

  • The reactor pattern: the mechanism at the heart of the Node.js asynchronous architecture

  • The Node.js callback pattern and its set of conventions

  • The module system and its patterns: the fundamental mechanisms for organizing code in Node.js

  • The observer pattern and its Node.js incarnation: the EventEmitter class

 

The Node.js philosophy


Every platform has its own philosophy—a set of principles and guidelines that are generally accepted by the community, an ideology of doing things that influences the evolution of a platform, and how applications are developed and designed. Some of these principles arise from the technology itself, some of them are enabled by its ecosystem, some are just trends in the community, and others are evolutions of different ideologies. In Node.js, some of these principles come directly from its creator, Ryan Dahl, from all the people who contributed to the core, from charismatic figures in the community, and some of the principles are inherited from the JavaScript culture or are influenced by the Unix philosophy.

None of these rules are imposed and they should always be applied with common sense; however, they can prove to be tremendously useful when we are looking for a source of inspiration while designing our programs.

Note

You can find an extensive list of software development philosophies in Wikipedia at http://en.wikipedia.org/wiki/List_of_software_development_philosophies.

Small core

The Node.js core itself has its foundations built on a few principles; one of these is, having the smallest set of functionality, leaving the rest to the so-called userland (or userspace), the ecosystem of modules living outside the core. This principle has an enormous impact on the Node.js culture, as it gives freedom to the community to experiment and iterate fast on a broader set of solutions within the scope of the userland modules, instead of being imposed with one slowly evolving solution that is built into the more tightly controlled and stable core. Keeping the core set of functionality to the bare minimum then, not only becomes convenient in terms of maintainability, but also in terms of the positive cultural impact that it brings on the evolution of the entire ecosystem.

Small modules

Node.js uses the concept of module as a fundamental mean to structure the code of a program. It is the brick for creating applications and reusable libraries called packages (a package is also frequently referred to as just module; since, usually it has one single module as an entry point). In Node.js, one of the most evangelized principles is to design small modules, not only in terms of code size, but most importantly in terms of scope.

This principle has its roots in the Unix philosophy, particularly in two of its precepts, which are as follows:

  • "Small is beautiful."

  • "Make each program do one thing well."

Node.js brought these concepts to a whole new level. Along with the help of npm, the official package manager, Node.js helps solving the dependency hell problem by making sure that each installed package will have its own separate set of dependencies, thus enabling a program to depend on a lot of packages without incurring in conflicts. The Node way, in fact, involves extreme levels of reusability, whereby applications are composed of a high number of small, well-focused dependencies. While this can be considered unpractical or even totally unfeasible in other platforms, in Node.js this practice is encouraged. As a consequence, it is not rare to find npm packages containing less than 100 lines of code or exposing only one single function.

Besides the clear advantage in terms of reusability, a small module is also considered to be the following:

  • Easier to understand and use

  • Simpler to test and maintain

  • Perfect to share with the browser

Having smaller and more focused modules empowers everyone to share or reuse even the smallest piece of code; it's the Don't Repeat Yourself (DRY) principle applied at a whole new level.

Small surface area

In addition to being small in size and scope, Node.js modules usually also have the characteristic of exposing only a minimal set of functionality. The main advantage here is an increased usability of the API, which means that the API becomes clearer to use and is less exposed to erroneous usage. Most of the time, in fact, the user of a component is interested only in a very limited and focused set of features, without the need to extend its functionality or tap into more advanced aspects.

In Node.js, a very common pattern for defining modules is to expose only one piece of functionality, such as a function or a constructor, while letting more advanced aspects or secondary features become properties of the exported function or constructor. This helps the user to identify what is important and what is secondary. It is not rare to find modules that expose only one function and nothing else, for the simple fact that it provides a single, unmistakably clear entry point.

Another characteristic of many Node.js modules is the fact that they are created to be used rather than extended. Locking down the internals of a module by forbidding any possibility of an extension might sound inflexible, but it actually has the advantage of reducing the use cases, simplifying its implementation, facilitating its maintenance, and increasing its usability.

Simplicity and pragmatism

Have you ever heard of the Keep It Simple, Stupid (KISS) principle? Or the famous quote:

 

"Simplicity is the ultimate sophistication."

 
 -- Leonardo da Vinci

Richard P. Gabriel, a prominent computer scientist coined the term worse is better to describe the model, whereby less and simpler functionality is a good design choice for software. In his essay, The rise of worse is better, he says:

"The design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design."

Designing a simple, as opposed to a perfect, feature-full software, is a good practice for several reasons: it takes less effort to implement, allows faster shipping with less resources, is easier to adapt, and is easier to maintain and understand. These factors foster the community contributions and allow the software itself to grow and improve.

In Node.js, this principle is also enabled by JavaScript, which is a very pragmatic language. It's not rare, in fact, to see simple functions, closures, and object literals replacing complex class hierarchies. Pure object-oriented designs often try to replicate the real world using the mathematical terms of a computer system without considering the imperfection and the complexity of the real world itself. The truth is that our software is always an approximation of the reality and we would probably have more success in trying to get something working sooner and with reasonable complexity, instead of trying to create a near-perfect software with a huge effort and tons of code to maintain.

Throughout this book, we will see this principle in action many times. For example, a considerable number of traditional design patterns, such as Singleton or Decorator can have a trivial, even if sometimes not foolproof implementation and we will see how an uncomplicated, practical approach most of the time is preferred to a pure, flawless design.

 

The reactor pattern


In this section, we will analyze the reactor pattern, which is the heart of the Node.js asynchronous nature. We will go through the main concepts behind the pattern, such as the single-threaded architecture and the non-blocking I/O, and we will see how this creates the foundation for the entire Node.js platform.

I/O is slow

I/O is definitely the slowest among the fundamental operations of a computer. Accessing the RAM is in the order of nanoseconds (10e-9 seconds), while accessing data on the disk or the network is in the order of milliseconds (10e-3 seconds). For the bandwidth, it is the same story; RAM has a transfer rate consistently in the order of GB/s, while disk and network varies from MB/s to, optimistically, GB/s. I/O is usually not expensive in terms of CPU, but it adds a delay between the moment the request is sent and the moment the operation completes. On top of that, we also have to consider the human factor; often, the input of an application comes from a real person, for example, the click of a button or a message sent in a real-time chat application, so the speed and frequency of I/O don't depend only on technical aspects, and they can be many orders of magnitude slower than the disk or network.

Blocking I/O

In traditional blocking I/O programming, the function call corresponding to an I/O request will block the execution of the thread until the operation completes. This can go from a few milliseconds, in case of a disk access, to minutes or even more, in case the data is generated from user actions, such as pressing a key. The following pseudocode shows a typical blocking read performed against a socket:

//blocks the thread until the data is available
data = socket.read();
//data is available
print(data);

It is trivial to notice that a web server that is implemented using blocking I/O will not be able to handle multiple connections in the same thread; each I/O operation on a socket will block the processing of any other connection. For this reason, the traditional approach to handle concurrency in web servers is to kick off a thread or a process (or to reuse one taken from a pool) for each concurrent connection that needs to be handled. This way, when a thread blocks for an I/O operation it will not impact the availability of the other requests, because they are handled in separate threads.

The following image illustrates this scenario:

The preceding image lays emphasis on the amount of time each thread is idle, waiting for new data to be received from the associated connection. Now, if we also consider that any type of I/O can possibly block a request, for example, while interacting with databases or with the filesystem, we soon realize how many times a thread has to block in order to wait for the result of an I/O operation. Unfortunately, a thread is not cheap in terms of system resources, it consumes memory and causes context switches, so having a long running thread for each connection and not using it for most of the time, is not the best compromise in terms of efficiency.

Non-blocking I/O

In addition to blocking I/O, most modern operating systems support another mechanism to access resources, called non-blocking I/O. In this operating mode, the system call always returns immediately without waiting for the data to be read or written. If no results are available at the moment of the call, the function will simply return a predefined constant, indicating that there is no data available to return at that moment.

For example, in Unix operating systems, the fcntl() function is used to manipulate an existing file descriptor to change its operating mode to non-blocking (with the O_NONBLOCK flag). Once the resource is in non-blocking mode, any read operation will fail with a return code, EAGAIN, in case the resource doesn't have any data ready to be read.

The most basic pattern for accessing this kind of non-blocking I/O is to actively poll the resource within a loop until some actual data is returned; this is called busy-waiting. The following pseudocode shows you how it's possible to read from multiple resources using non-blocking I/O and a polling loop:

resources = [socketA, socketB, pipeA];
while(!resources.isEmpty()) {
  for(i = 0; i < resources.length; i++) {
    resource = resources[i];
    //try to read
    var data = resource.read();
    if(data === NO_DATA_AVAILABLE)
      //there is no data to read at the moment
      continue;
    if(data === RESOURCE_CLOSED)
      //the resource was closed, remove it from the list
      resources.remove(i);
    else
      //some data was received, process it
      consumeData(data);
  }
}

You can see that, with this simple technique, it is already possible to handle different resources in the same thread, but it's still not efficient. In fact, in the preceding example, the loop will consume precious CPU only for iterating over resources that are unavailable most of the time. Polling algorithms usually result in a huge amount of wasted CPU time.

Event demultiplexing

Busy-waiting is definitely not an ideal technique for processing non-blocking resources, but luckily, most modern operating systems provide a native mechanism to handle concurrent, non-blocking resources in an efficient way; this mechanism is called synchronous event demultiplexer or event notification interface. This component collects and queues I/O events that come from a set of watched resources, and block until new events are available to process. The following is the pseudocode of an algorithm that uses a generic synchronous event demultiplexer to read from two different resources:

socketA, pipeB;
watchedList.add(socketA, FOR_READ);        //[1]
watchedList.add(pipeB, FOR_READ);
while(events = demultiplexer.watch(watchedList)) {    //[2]
  //event loop
  foreach(event in events) {          //[3]
    //This read will never block and will always return data
    data = event.resource.read();
    if(data === RESOURCE_CLOSED)
      //the resource was closed, remove it from the watched list
      demultiplexer.unwatch(event.resource);
    else
      //some actual data was received, process it
      consumeData(data);
  }
}

These are the important steps of the preceding pseudocode:

  1. The resources are added to a data structure, associating each one of them with a specific operation, in our example a read.

  2. The event notifier is set up with the group of resources to be watched. This call is synchronous and blocks until any of the watched resources is ready for a read. When this occurs, the event demultiplexer returns from the call and a new set of events is available to be processed.

  3. Each event returned by the event demultiplexer is processed. At this point, the resource associated with each event is guaranteed to be ready to read and to not block during the operation. When all the events are processed, the flow will block again on the event demultiplexer until new events are again available to be processed. This is called the event loop.

It's interesting to see that with this pattern, we can now handle several I/O operations inside a single thread, without using a busy-waiting technique. The following image shows us how a web server would be able to handle multiple connections using a synchronous event demultiplexer and a single thread:

The previous image helps us understand how concurrency works in a single-threaded application using a synchronous event demultiplexer and non-blocking I/O. We can see that using only one thread does not impair our ability to run multiple I/O bound tasks concurrently. The tasks are spread over time, instead of being spread across multiple threads. This has the clear advantage of minimizing the total idle time of the thread, as clearly shown in the image. This is not the only reason for choosing this model. To have only a single thread, in fact, also has a beneficial impact on the way programmers approach concurrency in general. Throughout the book, we will see how the absence of in-process race conditions and multiple threads to synchronize, allows us to use much simpler concurrency strategies.

In the next chapter, we will have the opportunity to talk more about the concurrency model of Node.js.

The reactor pattern

We can now introduce the reactor pattern, which is a specialization of the algorithm presented in the previous section. The main idea behind it is to have a handler (which in Node.js is represented by a callback function) associated with each I/O operation, which will be invoked as soon as an event is produced and processed by the event loop. The structure of the reactor pattern is shown in the following image:

This is what happens in an application using the reactor pattern:

  1. The application generates a new I/O operation by submitting a request to the Event Demultiplexer. The application also specifies a handler, which will be invoked when the operation completes. Submitting a new request to the Event Demultiplexer is a non-blocking call and it immediately returns the control back to the application.

  2. When a set of I/O operations completes, the Event Demultiplexer pushes the new events into the Event Queue.

  3. At this point, the Event Loop iterates over the items of the Event Queue.

  4. For each event, the associated handler is invoked.

  5. The handler, which is part of the application code, will give back the control to the Event Loop when its execution completes (5a). However, new asynchronous operations might be requested during the execution of the handler (5b), causing new operations to be inserted in the Event Demultiplexer (1), before the control is given back to the Event Loop.

  6. When all the items in the Event Queue are processed, the loop will block again on the Event Demultiplexer which will then trigger another cycle.

The asynchronous behavior is now clear: the application expresses the interest to access a resource at one point in time (without blocking) and provides a handler, which will then be invoked at another point in time when the operation completes.

Note

A Node.js application will exit automatically when there are no more pending operations in the Event Demultiplexer, and no more events to be processed inside the Event Queue.

We can now define the pattern at the heart of Node.js.

Note

Pattern (reactor): handles I/O by blocking until new events are available from a set of observed resources, and then reacting by dispatching each event to an associated handler.

The non-blocking I/O engine of Node.js – libuv

Each operating system has its own interface for the Event Demultiplexer: epoll on Linux, kqueue on Mac OS X, and I/O Completion Port API (IOCP) on Windows. Besides that, each I/O operation can behave quite differently depending on the type of the resource, even within the same OS. For example, in Unix, regular filesystem files do not support non-blocking operations, so, in order to simulate a non-blocking behavior, it is necessary to use a separate thread outside the Event Loop. All these inconsistencies across and within the different operating systems required a higher-level abstraction to be built for the Event Demultiplexer. This is exactly why the Node.js core team created a C library called libuv, with the objective to make Node.js compatible with all the major platforms and normalize the non-blocking behavior of the different types of resource; libuv today represents the low-level I/O engine of Node.js.

Besides abstracting the underlying system calls, libuv also implements the reactor pattern, thus providing an API for creating event loops, managing the event queue, running asynchronous I/O operations, and queuing other types of tasks.

Note

A great resource to learn more about libuv is the free online book created by Nikhil Marathe, which is available at http://nikhilm.github.io/uvbook/.

The recipe for Node.js

The reactor pattern and libuv are the basic building blocks of Node.js, but we need the following three other components to build the full platform:

  • A set of bindings responsible for wrapping and exposing libuv and other low-level functionality to JavaScript.

  • V8, the JavaScript engine originally developed by Google for the Chrome browser. This is one of the reasons why Node.js is so fast and efficient. V8 is acclaimed for its revolutionary design, its speed, and for its efficient memory management.

  • A core JavaScript library (called node-core) that implements the high-level Node.js API.

Finally, this is the recipe of Node.js, and the following image represents its final architecture:

 

The callback pattern


Callbacks are the materialization of the handlers of the reactor pattern and they are literally one of those imprints that give Node.js its distinctive programming style. Callbacks are functions that are invoked to propagate the result of an operation and this is exactly what we need when dealing with asynchronous operations. They practically replace the use of the return instruction that, as we know, always executes synchronously. JavaScript is a great language to represent callbacks, because as we know, functions are first class objects and can be easily assigned to variables, passed as arguments, returned from another function invocation, or stored into data structures. Also, closures are an ideal construct for implementing callbacks. With closures, we can in fact reference the environment in which a function was created, practically, we can always maintain the context in which the asynchronous operation was requested, no matter when or where its callback is invoked.

Note

If you need to refresh your knowledge about closures, you can refer to the article on the Mozilla Developer Network at https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Closures.

In this section, we will analyze this particular style of programming made of callbacks instead of the return instructions.

The continuation-passing style

In JavaScript, a callback is a function that is passed as an argument to another function and is invoked with the result when the operation completes. In functional programming, this way of propagating the result is called continuation-passing style, for brevity, CPS. It is a general concept, and it is not always associated with asynchronous operations. In fact, it simply indicates that a result is propagated by passing it to another function (the callback), instead of directly returning it to the caller.

Synchronous continuation-passing style

To clarify the concept, let's take a look at a simple synchronous function:

function add(a, b) {
  return a + b;
}

There is nothing special here; the result is passed back to the caller using the return instruction; this is also called direct style, and it represents the most common way of returning a result in synchronous programming. The equivalent continuation-passing style of the preceding function would be as follows:

function add(a, b, callback) {
  callback(a + b);
}

The add() function is a synchronous CPS function, which means that it will return a value only when the callback completes its execution. The following code demonstrates this statement:

console.log('before');
add(1, 2, function(result) {
  console.log('Result: ' + result);
});
console.log('after');

Since add() is synchronous, the previous code will trivially print the following:

before
Result: 3
after

Asynchronous continuation-passing style

Now, let's consider the case where the add() function is asynchronous, which is as follows:

function addAsync(a, b, callback) {
  setTimeout(function() {
    callback(a + b);
  }, 100);
}

In the previous code, we simply use setTimeout() to simulate an asynchronous invocation of the callback. Now, let's try to use this function and see how the order of the operations changes:

console.log('before');
addAsync(1, 2, function(result) {
  console.log('Result: ' + result);
});
console.log('after');

The preceding code will print the following:

before
after
Result: 3

Since setTimeout() triggers an asynchronous operation, it will not wait anymore for the callback to be executed, but instead, it returns immediately giving the control back to addAsync(), and then back to its caller. This property in Node.js is crucial, as it allows the stack to unwind, and the control to be given back to the event loop as soon as an asynchronous request is sent, thus allowing a new event from the queue to be processed.

The following image shows how this works:

When the asynchronous operation completes, the execution is then resumed starting from the callback provided to the asynchronous function that caused the unwinding. The execution will start from the Event Loop, so it will have a fresh stack. This is where JavaScript comes in really handy, in fact, thanks to closures it is trivial to maintain the context of the caller of the asynchronous function, even if the callback is invoked at a different point in time and from a different location.

Note

A synchronous function blocks until it completes its operations. An asynchronous function returns immediately and the result is passed to a handler (in our case, a callback) at a later cycle of the event loop.

Non continuation-passing style callbacks

There are several circumstances in which the presence of a callback argument might make you think that a function is asynchronous or is using a continuation-passing style; that's not always true, let's take, for example, the map() method of the Array object:

var result = [1, 5, 7].map(function(element) {
  return element – 1;
});

Clearly, the callback is just used to iterate over the elements of the array, and not to pass the result of the operation. In fact, the result is returned synchronously using a direct style. The intent of a callback is usually clearly stated in the documentation of the API.

Synchronous or asynchronous?

We have seen how the order of the instructions changes radically depending on the nature of a function - synchronous or asynchronous. This has strong repercussions on the flow of the entire application, both in correctness and efficiency. The following is an analysis of these two paradigms and their pitfalls. In general, what must be avoided, is creating inconsistency and confusion around the nature of an API, as doing so can lead to a set of problems which might be very hard to detect and reproduce. To drive our analysis, we will take as example the case of an inconsistently asynchronous function.

An unpredictable function

One of the most dangerous situations is to have an API that behaves synchronously under certain conditions and asynchronously under others. Let's take the following code as an example:

var fs = require('fs');
var cache = {};
function inconsistentRead(filename, callback) {
  if(cache[filename]) {
    //invoked synchronously
    callback(cache[filename]);  
  } else {
    //asynchronous function
    fs.readFile(filename, 'utf8', function(err, data) {
      cache[filename] = data;
      callback(data);
    });
  }
}

The preceding function uses the cache variable to store the results of different file read operations. Please bear in mind that this is just an example, it does not have error management, and the caching logic itself is suboptimal. Besides this, the preceding function is dangerous because it behaves asynchronously until the cache is not set—which is until the fs.readFile() function returns its results—but it will also be synchronous for all the subsequent requests for a file already in the cache—triggering an immediate invocation of the callback.

Unleashing Zalgo

Now, let's see how the use of an unpredictable function, such as the one that we defined previously, can easily break an application. Consider the following code:

function createFileReader(filename) {
  var listeners = [];
  inconsistentRead(filename, function(value) {
    listeners.forEach(function(listener) {
      listener(value);
    });
  });

  return {
    onDataReady: function(listener) {
      listeners.push(listener);
    }
  };
}

When the preceding function is invoked, it creates a new object that acts as a notifier, allowing to set multiple listeners for a file read operation. All the listeners will be invoked at once when the read operation completes and the data is available. The preceding function uses our inconsistentRead() function to implement this functionality. Let's now try to use the createFileReader() function:

var reader1 = createFileReader('data.txt');  
reader1.onDataReady(function(data) {
  console.log('First call data: ' + data);
  
  //...sometime later we try to read again from
  //the same file
  var reader2 = createFileReader('data.txt');
  reader2.onDataReady(function(data) {
    console.log('Second call data: ' + data);
  });
}); 

The preceding code will print the following output:

First call data: some data

As you can see, the callback of the second operation is never invoked. Let's see why:

  • During the creation of reader1, our inconsistentRead() function behaves asynchronously, because there is no cached result available. Therefore, we have all the time to register our listener, as it will be invoked later in another cycle of the event loop, when the read operation completes.

  • Then, reader2 is created in a cycle of the event loop in which the cache for the requested file already exists. In this case, the inner call to inconsistentRead() will be synchronous. So, its callback will be invoked immediately, which means that also all the listeners of reader2 will be invoked synchronously. However, we are registering the listeners after the creation of reader2, so they will never be invoked.

The callback behavior of our inconsistentRead() function is really unpredictable, as it depends on many factors, such as the frequency of its invocation, the filename passed as argument, and the amount of time taken to load the file.

The bug that we've just seen might be extremely complicated to identify and reproduce in a real application. Imagine to use a similar function in a web server, where there can be multiple concurrent requests; imagine seeing some of those requests hanging, without any apparent reason and without any error being logged. This definitely falls under the category of nasty defects.

Isaac Z. Schlueter, creator of npm and former Node.js project lead, in one of his blog posts compared the use of this type of unpredictable functions to unleashing Zalgo. If you're not familiar with Zalgo, you are invited to find out what it is.

Note

You can find the original Isaac Z. Schlueter's post at http://blog.izs.me/post/59142742143/designing-apis-for-asynchrony.

Using synchronous APIs

The lesson to learn from the unleashing Zalgo example is that it is imperative for an API to clearly define its nature, either synchronous or asynchronous.

One suitable fix for our inconsistentRead() function, is to make it totally synchronous. This is possible because Node.js provides a set of synchronous direct style APIs for most of the basic I/O operations. For example, we can use the fs.readFileSync() function in place of its asynchronous counterpart. The code would now be as follows:

var fs = require('fs');
var cache = {};
function consistentReadSync(filename) {
  if(cache[filename]) {
    return cache[filename];  
  } else {
    cache[filename] = fs.readFileSync(filename, 'utf8');
    return cache[filename];
  }
}

We can see that the entire function was also converted to a direct style. There is no reason for the function to have a continuation-passing style if it is synchronous. In fact, we can state that it is always a good practice to implement a synchronous API using a direct style; this will eliminate any confusion around its nature and will also be more efficient from a performance perspective.

Tip

Pattern: prefer the direct style for purely synchronous functions.

Please bear in mind that changing an API from CPS to a direct style, or from asynchronous to synchronous, or vice versa might also require a change to the style of all the code using it. For example, in our case, we will have to totally change the interface of our createFileReader() API and adapt it to work always synchronously.

Also, using a synchronous API instead of an asynchronous one has some caveats:

  • A synchronous API might not be always available for the needed functionality.

  • A synchronous API will block the event loop and put the concurrent requests on hold. It practically breaks the Node.js concurrency, slowing down the whole application. We will see later in the book what this really means for our applications.

In our consistentReadSync() function, the risk of blocking the event loop is partially mitigated, because the synchronous I/O API is invoked only once per each filename, while the cached value will be used for all the subsequent invocations. If we have a limited number of static files, then using consistentReadSync() won't have a big effect on our event loop. Things can change quickly if we have to read many files and only once. Using synchronous I/O in Node.js is strongly discouraged in many circumstances; however, in some situations, this might be the easiest and most efficient solution. Always evaluate your specific use case in order to choose the right alternative.

Tip

Use blocking API only when they don't affect the ability of the application to serve concurrent requests.

Deferred execution

Another alternative for fixing our inconsistentRead() function is to make it purely asynchronous. The trick here is to schedule the synchronous callback invocation to be executed "in the future" instead of being run immediately in the same event loop cycle. In Node.js, this is possible using process.nextTick(), which defers the execution of a function until the next pass of the event loop. Its functioning is very simple; it takes a callback as an argument and pushes it on the top of the event queue, in front of any pending I/O event, and returns immediately. The callback will then be invoked as soon as the event loop runs again.

Let's apply this technique to fix our inconsistentRead() function as follows:

var fs = require('fs');
var cache = {};
function consistentReadAsync(filename, callback) {
  if(cache[filename]) {
    process.nextTick(function() {
      callback(cache[filename]);
    });
  } else {
    //asynchronous function
    fs.readFile(filename, 'utf8', function(err, data) {
      cache[filename] = data;
      callback(data);
    });
  }
}

Now, our function is guaranteed to invoke its callback asynchronously, under any circumstances.

Another API for deferring the execution of code is setImmediate(), which—despite the name—might actually be slower than process.nextTick(). While their purpose is very similar, their semantic is quite different. Callbacks deferred with process.nextTick() run before any other I/O event is fired, while with setImmediate(), the execution is queued behind any I/O event that is already in the queue. Since process.nextTick() runs before any already scheduled I/O, it might cause I/O starvation under certain circumstances, for example, a recursive invocation; this can never happen with setImmediate(). We will learn to appreciate the difference between these two APIs when we analyze the use of deferred invocation for running synchronous CPU-bound tasks later in the book.

Tip

Pattern: we guarantee that a callback is invoked asynchronously by deferring its execution using process.nextTick().

Node.js callback conventions

In Node.js, continuation-passing style APIs and callbacks follow a set of specific conventions. These conventions apply to the Node.js core API but they are also followed virtually by every userland module and application. So, it's very important that we understand them and make sure that we comply whenever we need to design an asynchronous API.

Callbacks come last

In Node.js, if a function accepts in input a callback, this has to be passed as the last argument. Let's take the following Node.js core API as an example:

fs.readFile(filename, [options], callback)

As you can see from the signature of the preceding function, the callback is always put in last position, even in the presence of optional arguments. The motivation for this convention is that the function call is more readable in case the callback is defined in place.

Error comes first

In CPS, errors are propagated as any other type of result, which means using the callback. In Node.js, any error produced by a CPS function is always passed as the first argument of the callback, and any actual result is passed starting from the second argument. If the operation succeeds without errors, the first argument will be null or undefined. The following code shows you how to define a callback complying with this convention:

fs.readFile('foo.txt', 'utf8', function(err, data) {
  if(err)
    handleError(err);
  else
    processData(data);
});

It is a good practice to always check for the presence of an error, as not doing so will make it harder for us to debug our code and discover the possible points of failures. Another important convention to take into account is that the error must always be of type Error. This means that simple strings or numbers should never be passed as error objects.

Propagating errors

Propagating errors in synchronous, direct style functions is done with the well-known throw command, which causes the error to jump up in the call stack until it's caught.

In asynchronous CPS however, proper error propagation is done by simply passing the error to the next callback in the CPS chain. The typical pattern looks as follows:

var fs = require('fs');
function readJSON(filename, callback) {
  fs.readFile(filename, 'utf8', function(err, data) {
    var parsed;
    if(err)
      //propagate the error and exit the current function
      return callback(err);

    try {
      //parse the file contents
      parsed = JSON.parse(data);
    } catch(err) {
      //catch parsing errors
      return callback(err);
    }
    //no errors, propagate just the data
    callback(null, parsed);
  });
};

The detail to notice in the previous code is how the callback is invoked when we want to pass a valid result and when we want to propagate an error.

Uncaught exceptions

You might have seen from the readJSON() function defined previously that in order to avoid any exception to be thrown into the fs.readFile() callback, we put a try-catch block around JSON.parse(). Throwing inside an asynchronous callback, in fact, will cause the exception to jump up to the event loop and never be propagated to the next callback.

In Node.js, this is an unrecoverable state and the application will simply shut down printing the error to the stderr interface. To demonstrate this, let's try to remove the try-catch block from the readJSON() function defined previously:

var fs = require('fs');
function readJSONThrows(filename, callback) {
  fs.readFile(filename, 'utf8', function(err, data) {
    if(err)
      return callback(err);
    //no errors, propagate just the data
    callback(null, JSON.parse(data));
  });
};

Now, in the function we just defined, there is no way of catching an eventual exception coming from JSON.parse(). Let's try, for example, to parse an invalid JSON file with the following code:

readJSONThrows('nonJSON.txt', function(err) {
  console.log(err);
});

This would result in the application being abruptly terminated and the following exception being printed on the console:

SyntaxError: Unexpected token d
    at Object.parse (native)
    at [...]/06_uncaught_exceptions/uncaught.js:7:25
    at fs.js:266:14
    at Object.oncomplete (fs.js:107:15)

Now, if we look at the preceding stack trace, we will see that it starts somewhere from the fs.js module, practically from the point at which the native API has completed reading and returned its result back to the fs.readFile() function, via the event loop. This clearly shows us that the exception traveled from our callback into the stack that we saw, and then straight into the event loop, where it's finally caught and thrown in the console.

This also means that wrapping the invocation of readJSONThrows() with a try-catch block will not work, because the stack in which the block operates is different from the one in which our callback is invoked. The following code shows the anti-pattern that we just described:

try {
  readJSONThrows('nonJSON.txt', function(err, result) {
    [...]
  });
} catch(err) {
  console.log('This will not catch the JSON parsing exception');
}

The preceding catch statement will never receive the JSON parsing exception, as it will travel back to the stack in which the exception was thrown, and we just saw that the stack ends up in the event loop and not with the function that triggers the asynchronous operation.

We already said that the application is aborted the moment an exception reaches the event loop; however, we still have a last chance to perform some cleanup or logging before the application terminates. In fact, when this happens, Node.js emits a special event called uncaughtException just before exiting the process. The following code shows a sample use case:

process.on('uncaughtException', function(err){
  console.error('This will catch at last the ' +
    'JSON parsing exception: ' + err.message);
  //without this, the application would continue
  process.exit(1);
});

It's important to understand that an uncaught exception leaves the application in a state that is not guaranteed to be consistent, which can lead to unforeseeable problems. For example, there might still have incomplete I/O requests running, or closures might have become inconsistent. That's why it is always advised, especially in production, to exit anyway from the application after an uncaught exception is received.

 

The module system and its patterns


Modules are the bricks for structuring non-trivial applications, but also the main mechanism to enforce information hiding by keeping private all the functions and variables that are not explicitly marked to be exported. In this section, we will introduce the Node.js module system and its most common usage patterns.

The revealing module pattern

One of the major problems with JavaScript is the absence of namespacing. Programs run in the global scope polluting it with data that comes from both internal application code and dependencies. A popular technique to solve this problem is called revealing module pattern and it looks like the following:

var module = (function() {
  var privateFoo = function() {...};
  var privateVar = [];

  var export = {
    publicFoo: function() {...},
    publicBar: function() {...}
  }

  return export;
})();

This pattern leverages a self-invoking function to create a private scope, exporting only the parts that are meant to be public. In the preceding code, the module variable contains only the exported API, while the rest of the module content is practically inaccessible from outside. As we will see in a moment, the idea behind this pattern is used as a base for the Node.js module system.

Node.js modules explained

CommonJS is a group with the aim to standardize the JavaScript ecosystem, and one of their most popular proposals is called CommonJS modules. Node.js built its module system on top of this specification, with the addition of some custom extensions. To describe how it works, we can make an analogy with the revealing module pattern, where each module runs in a private scope, so that every variable that is defined locally does not pollute the global namespace.

A homemade module loader

To explain how this works, let's build a similar system from scratch. The code that follows creates a function that mimics a subset of the functionality of the original require() function of Node.js.

Let's start by creating a function that loads the content of a module, wraps it into a private scope, and evaluates it:

function loadModule(filename, module, require) {
  var wrappedSrc =
    '(function(module, exports, require) {' +
      fs.readFileSync(filename, 'utf8') +
    '})(module, module.exports, require);';
  eval(wrappedSrc);
}

The source code of a module is essentially wrapped into a function, as it was for the revealing module pattern. The difference here is that we pass a list of variables to the module, in particular: module, exports, and require. Make a note of how the exports argument of the wrapping function is initialized with the contents of module.exports, as we will talk about this later.

Note

Please bear in mind that this is only an example and you will rarely need to evaluate some source code in a real application. Features such as eval() or the functions of the vm module (http://nodejs.org/api/vm.html) can be easily used in the wrong way or with the wrong input, thus opening a system to code injection attacks. They should always be used with extreme care or avoided altogether.

Let's now see what these variables contain by implementing our require() function:

var require = function(moduleName) {
  console.log('Require invoked for module: ' + moduleName);
  var id = require.resolve(moduleName);      //[1]
  if(require.cache[id]) {           //[2]
    return require.cache[id].exports;
  }
 
  //module metadata
  var module = {               //[3]
    exports: {},
    id: id
  };
  //Update the cache
  require.cache[id] = module;           //[4]

  //load the module
  loadModule(id, module, require);         //[5]
  
  //return exported variables
  return module.exports;             //[6]
};
require.cache = {};
require.resolve = function(moduleName) {
  /* resolve a full module id from the moduleName */
}

The preceding function simulates the behavior of the original require() function of Node.js, which is used to load a module. Of course, this is just for educative purposes and it does not accurately or completely reflect the internal behavior of the real require() function, but it's great to understand the internals of the Node.js module system, how a module is defined, and loaded. What our homemade module system does is explained as follows:

  1. A module name is accepted as input and the very first thing that we do is resolve the full path of the module, which we call id. This task is delegated to require.resolve(), which implements a specific resolving algorithm (we will talk about it later).

  2. If the module was already loaded in the past, it should be available in the cache. In this case, we just return it immediately.

  3. If the module was not yet loaded, we set up the environment for the first load. In particular, we create a module object that contains an exports property initialized with an empty object literal. This property will be used by the code of the module to export any public API.

  4. The module object is cached.

  5. The module source code is read from its file and the code is evaluated, as we have seen before. We provide to the module, the module object that we just created, and a reference to the require() function. The module exports its public API by manipulating or replacing the module.exports object.

  6. Finally, the content of module.exports, which represents the public API of the module, is returned to the caller.

As we see, there is nothing magical behind the workings of the Node.js module system; the trick is all in the wrapper we create around a module's source code and the artificial environment in which we run it.

Defining a module

By looking at how our homemade require() function works, we should now know how to define a module. The following code gives us an example:

//load another dependency
var dependency = require('./anotherModule');

//a private function
function log() {
  console.log('Well done ' + dependency.username);
}

//the API to be exported for public use
module.exports.run = function() {
  log();
};

The essential concept to remember is that everything inside a module is private unless it's assigned to the module.exports variable. The contents of this variable are then cached and returned when the module is loaded using require().

Defining globals

Even if all the variables and functions that are declared in a module are defined in its local scope, it is still possible to define a global variable. In fact, the module system exposes a special variable called global, which can be used for this purpose. Everything that is assigned to this variable will end up automatically in the global scope.

Note

Please note that polluting the global scope is considered a bad practice and nullifies the advantage of having a module system. So, use it only if you really know what you are doing.

module.exports vs exports

For many developers who are not yet familiar with Node.js, a common source of confusion is the difference between using exports and module.exports to expose a public API. The code of our homemade require function should again clear any doubt. The variable exports is just a reference to the initial value of module.exports; we have seen that such a value is essentially a simple object literal created before the module is loaded.

This means that we can only attach new properties to the object referenced by the exports variable, as shown in the following code:

exports.hello = function() {
 console.log('Hello');
}

Reassigning the exports variable doesn't have any effect, because it doesn't change the contents of module.exports, it will only reassign the variable itself. The following code is therefore wrong:

exports = function() {
 console.log('Hello');
}

If we want to export something other than an object literal, as for example a function, an instance, or even a string, we have to reassign module.exports as follows:

module.exports = function() {
 console.log('Hello');
}

require is synchronous

Another important detail that we should take into account is that our homemade require function is synchronous. In fact, it returns the module contents using a simple direct style, and no callback is required. This is true for the original Node.js require() function too. As a consequence, any assignment to module.export must be synchronous as well. For example, the following code is incorrect:

setTimeout(function() {
  module.exports = function() {...};
}, 100);

This property has important repercussions in the way we define modules, as it limits us to mostly using synchronous code during the definition of a module. This is actually one of the most important reasons why the core Node.js libraries offer synchronous APIs as an alternative to most of the asynchronous ones.

If we need some asynchronous initialization steps for a module, we can always define and export an uninitialized module that is initialized asynchronously at a later time. The problem with this approach though, is that loading such a module using require does not guarantee that it's ready to be used. In Chapter 6, Recipes, we will analyze this problem in detail and we will present some patterns to solve this issue elegantly.

Note

For the sake of curiosity, you might want to know that in its early days, Node.js used to have an asynchronous version of require(), but it was soon removed because it was overcomplicating a functionality that was actually meant to be used only at initialization time, and where asynchronous I/O brings more complexities than advantages.

The resolving algorithm

The term dependency hell, describes a situation whereby the dependencies of a software, in turn depend on a shared dependency, but require different incompatible versions. Node.js solves this problem elegantly by loading a different version of a module depending on where the module is loaded from. All the merits of this feature go to npm and also to the resolving algorithm used in the require function.

Let's now give a quick overview of this algorithm. As we saw, the resolve() function takes a module name (which we will call here, moduleName) as input and it returns the full path of the module. This path is then used to load its code and also to identify the module uniquely. The resolving algorithm can be divided into the following three major branches:

  • File modules: If moduleName starts with "/" it's considered already an absolute path to the module and it's returned as it is. If it starts with "./", then moduleName is considered a relative path, which is calculated starting from the requiring module.

  • Core modules: If moduleName is not prefixed with "/" or "./", the algorithm will first try to search within the core Node.js modules.

  • Package modules: If no core module is found matching moduleName, then the search continues by looking for a matching module into the first node_modules directory that is found navigating up in the directory structure starting from the requiring module. The algorithm continues to search for a match by looking into the next node_modules directory up in the directory tree, until it reaches the root of the filesystem.

For file and package modules, both the individual files and directories can match moduleName. In particular, the algorithm will try to match the following:

  • <moduleName>.js

  • <moduleName>/index.js

  • The directory/file specified in the main property of <moduleName>/package.json

Note

The complete, formal documentation of the resolving algorithm can be found at http://nodejs.org/api/modules.html#modules_all_together.

The node_modules directory is actually where npm installs the dependencies of each package. This means that, based on the algorithm we just described, each package can have its own private dependencies. For example, consider the following directory structure:

myApp
├── foo.js
└── node_modules
    ├── depA
    │   └── index.js
    ├── depB
    │   ├── bar.js
    │   └── node_modules
    │       └── depA
    │           └── index.js
    └── depC
        ├── foobar.js
        └── node_modules
            └── depA
                └── index.js

In the preceding example, myApp, depB, and depC all depend on depA; however, they all have their own private version of the dependency! Following the rules of the resolving algorithm, using require('depA') will load a different file depending on the module that requires it, for example:

  • Calling require('depA') from /myApp/foo.js will load /myApp/node_modules/depA/index.js

  • Calling require('depA') from /myApp/node_modules/depB/bar.js will load /myApp/node_modules/depB/node_modules/depA/index.js

  • Calling require('depA') from /myApp/node_modules/depC/foobar.js will load /myApp/node_modules/depC/node_modules/depA/index.js

The resolving algorithm is the magic behind the robustness of the Node.js dependency management, and is what makes it possible to have hundreds or even thousands of packages in an application without having collisions or problems of version compatibility.

Note

The resolving algorithm is applied transparently for us when we invoke require(); however, if needed, it can still be used directly by any module by simply invoking require.resolve().

The module cache

Each module is loaded and evaluated only the first time it is required, since any subsequent call of require() will simply return the cached version. This should result clear by looking at the code of our homemade require function. Caching is crucial for performances, but it also has some important functional implications:

  • It makes it possible to have cycles within module dependencies

  • It guarantees, to some extent, that always the same instance is returned when requiring the same module from within a given package

The module cache is exposed in the require.cache variable, so it is possible to directly access it if needed. A common use case is to invalidate any cached module by deleting the relative key in the require.cache variable, a practice very useful during testing but very dangerous if applied in normal circumstances.

Cycles

Many consider circular dependencies as an intrinsic design issue, but it is something which might actually happen in a real project, so it's useful for us to know at least how this works in Node.js. If we look again at our homemade require() function, we immediately get a glimpse of how this might work and what are its caveats.

Suppose we have two modules defined as follows:

  • Module a.js:

    exports.loaded = false;
    var b = require('./b');
    module.exports = {
      bWasLoaded: b.loaded,
      loaded: true
    };
  • Module b.js:

    exports.loaded = false;
    var a = require('./a');
    module.exports = {
      aWasLoaded: a.loaded,
      loaded: true
    };

Now, let's try to load these from another module, main.js, as follows:

var a = require('./a');
var b = require('./b');
console.log(a);
console.log(b);

The preceding code will print the following output:

{ bWasLoaded: true, loaded: true }
{ aWasLoaded: false, loaded: true }

This result reveals the caveats of circular dependencies. While both the modules are completely initialized the moment they are required from the main module, the a.js module will be incomplete when it is loaded from b.js. In particular, its state will be the one that it reached the moment it required b.js. This behavior should ring another bell, which will be confirmed if we swap the order in which the two modules are required in main.js.

If you try it, you will see that this time it will be the module a.js that will receive an incomplete version of b.js. We understand now that this can become quite a fuzzy business if we lose control of which module is loaded first, which can happen quite easily if the project is big enough.

Module definition patterns

The module system, besides being a mechanism for loading dependencies, is also a tool for defining APIs. As for any other problem related to API design, the main factor to consider is the balance between private and public functionality. The aim is to maximize information hiding and API usability, while balancing these with other software qualities like extensibility and code reuse.

In this section, we will analyze some of the most popular patterns for defining modules in Node.js; each one has its own balance of information hiding, extensibility, and code reuse.

Named exports

The most basic method for exposing a public API is using named exports, which consists in assigning all the values we want to make public to properties of the object referenced by exports (or module.exports). In this way, the resulting exported object becomes a container or namespace for a set of related functionality.

The following code shows a module implementing this pattern:

//file logger.js
exports.info = function(message) {
  console.log('info: ' + message);
};

exports.verbose = function(message) {
  console.log('verbose: ' + message);
};

The exported functions are then available as properties of the loaded module, as shown in the following code:

//file main.js
var logger = require('./logger');
logger.info('This is an informational message');
logger.verbose('This is a verbose message');

Most of the Node.js core modules use this pattern.

Note

The CommonJS specification only allows the use of the exports variable to expose public members. Therefore, the named exports pattern is the only one that is really compatible with the CommonJS specification. The use of module.exports is an extension provided by Node.js to support a broader range of module definition patterns, as those we are going to see next.

Exporting a function

One of the most popular module definition patterns consists in reassigning the whole module.exports variable to a function. Its main strength it's the fact that it exposes only a single functionality, which provides a clear entry point for the module, and makes it simple to understand and use; it also honors the principle of small surface area very well. This way of defining modules is also known in the community as substack pattern, after one of its most prolific adopters, James Halliday (nickname substack). The following code is an example of this pattern:

//file logger.js

module.exports = function(message) {
  console.log('info: ' + message);
};

A possible extension of this pattern is using the exported function as namespace for other public APIs. This is a very powerful combination, because it still gives the module the clarity of a single entry point (the main exported function), but it also allows us to expose other functionalities that have secondary or more advanced use cases. The following code shows you how to extend the module we defined previously by using the exported function as a namespace:

module.exports.verbose = function(message) {
  console.log('verbose: ' + message);
};

The following code demonstrates how to use the module that we just defined:

//file main.js
var logger = require('./logger');
logger('This is an informational message');
logger.verbose('This is a verbose message');

Even though exporting just a function might seem a limitation, in reality, it's a perfect way to put the emphasis on a single functionality—the most important for the module—while giving less visibility to secondary aspects, which are instead exposed as properties of the exported function itself.

Note

Pattern (substack): expose the main functionality of a module by exporting only one function. Use the exported function as namespace to expose any auxiliary functionality.

Exporting a constructor

A module that exports a constructor is a specialization of a module that exports a function. The difference is that with this new pattern, we allow the user to create new instances using the constructor, but we also give them the ability to extend its prototype and forge new classes. The following is an example of this pattern:

//file logger.js
function Logger(name) {
  this.name = name;
};
Logger.prototype.log = function(message) {
  console.log('[' + this.name + '] ' + message);
};
Logger.prototype.info = function(message) {
  this.log('info: ' + message);
};
Logger.prototype.verbose = function(message) {
  this.log('verbose: ' + message);
};
module.exports = Logger;

And, we can use the preceding module as follows:

//file logger.js
var Logger = require('./logger');
var dbLogger = new Logger('DB');
dbLogger.info('This is an informational message');
var accessLogger = new Logger('ACCESS');
accessLogger.verbose('This is a verbose message');

Exporting a constructor still provides a single entry point for the module, but compared to the substack pattern, it exposes a lot more of the module internals; however on the other side it allows much more power when it comes to extending its functionality.

A variation of this pattern consists in applying a guard against invocations that don't use the new instruction. This little trick allows us to use our module as a factory. The following code shows you how this works:

function Logger(name) {
  if(!(this instanceof Logger)) {
    return new Logger(name);
  }
  this.name = name;
};

The trick is simple; we check whether this exists and is an instance of Logger. If any of these conditions is false, it means that the Logger() function was invoked without using new, so we proceed with creating the new instance properly and returning it to the caller. This technique allows us to use the module also as a factory, as shown in the following code:

//file logger.js
var Logger = require('./logger');
var dbLogger = Logger('DB');
accessLogger.verbose('This is a verbose message');

Exporting an instance

We can leverage the caching mechanism of require() to easily define stateful instances—objects with a state created from a constructor or a factory, which can be shared across different modules. The following code shows an example of this pattern:

//file logger.js
function Logger(name) {
  this.count = 0;
  this.name = name;
};
Logger.prototype.log = function(message) {
  this.count++;
  console.log('[' + this.name + '] ' + message);
};
module.exports = new Logger('DEFAULT');

This newly defined module can then be used as follows:

//file main.js
var logger = require('./logger');
logger.log('This is an informational message');

Because the module is cached, every module that requires the logger module will actually always retrieve the same instance of the object, thus sharing its state. This pattern is very much like creating a Singleton, however, it does not guarantee the uniqueness of the instance across the entire application, as it happens in the traditional Singleton pattern. When analyzing the resolving algorithm, we have seen in fact, that a module might be installed multiple times inside the dependency tree of an application. This results with multiple instances of the same logical module, all running in the context of the same Node.js application. In Chapter 5, Wiring Modules, we will analyze the consequences of exporting stateful instances and some of the patterns we can use as alternatives.

An extension to the pattern we just described, consists in exposing the constructor used to create the instance, in addition to the instance itself. This allows the user to create new instances of the same object, or even to extend it if necessary. To enable this, we just need to assign a new property to the instance, as shown in the following line of code:

module.exports.Logger = Logger;

Then, we can use the exported constructor to create other instances of the class, as follows:

var customLogger = new logger.Logger('CUSTOM');
customLogger.log('This is an informational message');

From a usability perspective, this is similar to using an exported function as namespace; the module exports the default instance of an object—the piece of functionality we might want to use most of the time—while more advanced features, such as the ability to create new instances or extend the object, are still made available through less exposed properties.

Modifying other modules or the global scope

A module can even export nothing. This can look a bit out of place, however, we should not forget that a module can modify the global scope and any object in it, including other modules in the cache. Please note that these are in general considered bad practices, but since this pattern can be useful and safe under some circumstances (for example, for testing) and is sometimes used in the wild, it is worth to know and understand it. So, we said a module can modify other modules or objects in the global scope. Well, this is called monkey patching, which generally refers to the practice of modifying the existing objects at runtime to change or extend their behavior or to apply temporary fixes.

The following example shows you how we can add a new function to another module:

//file patcher.js

// ./logger is another module
require('./logger').customMessage = function() {
  console.log('This is a new functionality');
};

Using our new patcher module would be as easy as writing the following code:

//file main.js

require('./patcher');
var logger = require('./logger');
logger.customMessage();

In the preceding code, patcher must be required before using the logger module for the first time in order to allow the patch to be applied.

The techniques described here are all dangerous ones to apply. The main concern is that, to have a module that modifies the global namespace or other modules is an operation with side effects. In other words, it affects the state of entities outside their scope, which can have consequences that are not always predictable, especially when multiple modules interact with the same entities. Imagine to have two different modules trying to set the same global variable, or modifying the same property of the same module; the effects might be unpredictable (which module wins?), but most importantly it would have repercussions on the entire application.

 

The observer pattern


Another important and fundamental pattern used in Node.js is the observer pattern. Together with reactor, callbacks, and modules, this is one of the pillars of the platform and an absolute prerequisite for using many node-core and userland modules.

Observer is an ideal solution for modeling the reactive nature of Node.js, and a perfect complement for callbacks. Let's give a formal definition as follows:

Note

Pattern (observer): defines an object (called subject), which can notify a set of observers (or listeners), when a change in its state happens.

The main difference from the callback pattern is that the subject can actually notify multiple observers, while a traditional continuation-passing style callback will usually propagate its result to only one listener, the callback.

The EventEmitter

In traditional object-oriented programming, the observer pattern requires interfaces, concrete classes, and a hierarchy; in Node.js, all becomes much simpler. The observer pattern is already built into the core and is available through the EventEmitter class. The EventEmitter class allows us to register one or more functions as listeners, which will be invoked when a particular event type is fired. The following image visually explains the concept:

The EventEmitter is a prototype, and it is exported from the events core module. The following code shows how we can obtain a reference to it:

var EventEmitter = require('events').EventEmitter;
var eeInstance = new EventEmitter();

The essential methods of the EventEmitter are given as follows:

  • on(event, listener): This method allows you to register a new listener (a function) for the given event type (a string)

  • once(event, listener): This method registers a new listener, which is then removed after the event is emitted for the first time

  • emit(event, [arg1], […]): This method produces a new event and provides additional arguments to be passed to the listeners

  • removeListener(event, listener): This method removes a listener for the specified event type

All the preceding methods will return the EventEmitter instance to allow chaining. The listener function has the signature, function([arg1], […]), so it simply accepts the arguments provided the moment the event is emitted. Inside the listener, this refers to the instance of the EventEmitter that produces the event.

We can already see that there is a big difference between a listener and a traditional Node.js callback; in particular, the first argument is not an error, but it can be any data passed to emit() at the moment of its invocation.

Create and use an EventEmitter

Let's see how we can use an EventEmitter in practice. The simplest way is to create a new instance and use it directly. The following code shows a function, which uses an EventEmitter to notify its subscribers in real time when a particular pattern is found in a list of files:

var EventEmitter = require('events').EventEmitter;
var fs = require('fs');

function findPattern(files, regex) {
  var emitter = new EventEmitter();
  files.forEach(function(file) {
    fs.readFile(file, 'utf8', function(err, content) {
      if(err)
        return emitter.emit('error', err);
      
      emitter.emit('fileread', file);
      var match = null;
      if(match = content.match(regex))
        match.forEach(function(elem) {
          emitter.emit('found', file, elem);
        });
    });
  });
  return emitter;
}

The EventEmitter created by the preceding function will produce the following three events:

  • fileread: This event occurs when a file is read

  • found: This event occurs when a match has been found

  • error: This event occurs when an error has occurred during the reading of the file

Let's see now how our findPattern() function can be used:

findPattern(
    ['fileA.txt', 'fileB.json'],
    /hello \w+/g
  )
  .on('fileread', function(file) {
    console.log(file + ' was read');
  })
  .on('found', function(file, match) {
    console.log('Matched "' + match + '" in file ' + file);
  })
  .on('error', function(err) {
    console.log('Error emitted: ' + err.message);
  });

In the preceding example, we registered a listener for each of the three event types produced by the EventEmitter which was created by our findPattern() function.

Propagating errors

The EventEmitter - as it happens for callbacks - cannot just throw exceptions when an error condition occurs, as they would be lost in the event loop if the event is emitted asynchronously. Instead, the convention is to emit a special event, called error, and to pass an Error object as an argument. That's exactly what we are doing in the findPattern() function that we defined earlier.

Note

It is always a good practice to register a listener for the error event, as Node.js will treat it in a special way and will automatically throw an exception and exit from the program if no associated listener is found.

Make any object observable

Sometimes, creating a new observable object directly from the EventEmitter class is not enough, as this makes it impractical to provide functionality that goes beyond the mere production of new events. It is more common, in fact, to have the need to make a generic object observable; this is possible by extending the EventEmitter class.

To demonstrate this pattern, let's try to implement the functionality of the findPattern() function in an object as follows:

var EventEmitter = require('events').EventEmitter;
var util = require('util');
var fs = require('fs');

function FindPattern(regex) {
  EventEmitter.call(this);
  this.regex = regex;
  this.files = [];
}
util.inherits(FindPattern, EventEmitter);

FindPattern.prototype.addFile = function(file) {
  this.files.push(file);
  return this;
};

FindPattern.prototype.find = function() {
  var self = this;
  self.files.forEach(function(file) {
    fs.readFile(file, 'utf8', function(err, content) {
      if(err)
        return self.emit('error', err);
      
      self.emit('fileread', file);
      var match = null;
      if(match = content.match(self.regex))
        match.forEach(function(elem) {
          self.emit('found', file, elem);
        });
    });
  });
  return this;
};

The FindPattern prototype that we defined extends the EventEmitter using the inherits() function provided by the core module util. This way, it becomes a full-fledged observable class. The following is an example of its usage:

var findPatternObject = new FindPattern(/hello \w+/);
findPatternObject
  .addFile('fileA.txt')
  .addFile('fileB.json')
  .find()
  .on('found', function(file, match) {
    console.log('Matched "' + match + '" in file ' + file);
  })
  .on('error', function(err) {
    console.log('Error emitted ' + err.message);
  });

We can now see how the FindPattern object has a full set of methods, in addition to being observable by inheriting the functionality of the EventEmitter.

This is a pretty common pattern in the Node.js ecosystem, for example, the Server object of the core http module defines methods such as listen(), close(), setTimeout(), and internally it also inherits from the EventEmitter function, thus allowing it to produce events, such as request, when a new request is received, or connection, when a new connection is established, or closed, when the server is closed.

Other notable examples of objects extending the EventEmitter are Node.js streams. We will analyze streams in more detail in Chapter 3, Coding with Streams.

Synchronous and asynchronous events

As with callbacks, events can be emitted synchronously or asynchronously, and it is crucial that we never mix the two approaches in the same EventEmitter, but even more importantly, when emitting the same event type, to avoid to produce the same problems that we described in the Unleashing Zalgo section.

The main difference between emitting synchronous or asynchronous events lies in the way listeners can be registered. When the events are emitted asynchronously, the user has all the time to register new listeners even after the EventEmitter is initialized, because the events are guaranteed not to be fired until the next cycle of the event loop. That's exactly what is happening in the findPattern() function. We defined this function previously and it represents a common approach that is used in most Node.js modules.

On the contrary, emitting events synchronously requires that all the listeners are registered before the EventEmitter function starts to emit any event. Let's look at an example:

function SyncEmit() {
  this.emit('ready');
}
util.inherits(SyncEmit, EventEmitter);

var syncEmit = new SyncEmit();
syncEmit.on('ready', function() {
  console.log('Object is ready to be used');
});

If the ready event was emitted asynchronously, then the previous code would work perfectly; however, the event is produced synchronously and the listener is registered after the event was already sent, so the result is that the listener is never invoked; the code will print nothing to the console.

Contrarily to callbacks, there are situations where using an EventEmitter in a synchronous fashion makes sense, given its different purpose. For this reason, it's very important to clearly highlight the behavior of our EventEmitter in its documentation to avoid confusion, and potentially a wrong usage.

EventEmitter vs Callbacks

A common dilemma when defining an asynchronous API is to check whether to use an EventEmitter or simply accept a callback. The general differentiating rule is semantic: callbacks should be used when a result must be returned in an asynchronous way; events should instead be used when there is a need to communicate that something has just happened.

But besides this simple principle, a lot of confusion is generated from the fact that the two paradigms are most of the time equivalent and allow you to achieve the same results. Consider the following code for an example:

function helloEvents() {
  var eventEmitter = new EventEmitter();
  setTimeout(function() {
    eventEmitter.emit('hello', 'world');
  }, 100);
  return eventEmitter;
}
function helloCallback(callback) {
  setTimeout(function() {
    callback('hello', 'world');
  }, 100);
}

The two functions helloEvents() and helloCallback() can be considered equivalent in terms of functionality; the first communicates the completion of the timeout using an event, the second uses a callback to notify the caller instead, passing the event type as an argument. But what really differentiates them is the readability, the semantic, and the amount of code that is required to be implemented or used. While we cannot give a deterministic set of rules to choose between one or the other style, we can certainly provide some hints to help take the decision.

As a first observation, we can say that callbacks have some limitations when it comes to supporting different types of events. In fact, we can still differentiate between multiple events by passing the type as an argument of the callback, or by accepting several callbacks, one for each supported event. However, this cannot exactly be considered an elegant API. In this situation, an EventEmitter can give a better interface and leaner code.

Another case where the EventEmitter might be preferable is when the same event can occur multiple times, or not occur at all. A callback, in fact, is expected to be invoked exactly once, whether the operation is successful or not. The fact that we have a possibly repeating circumstance should let us think again about the semantic nature of the occurrence, which is more similar to an event that has to be communicated rather than a result; in this case an EventEmitter is the preferred choice.

Lastly, an API using callbacks can notify only that particular callback, while using an EventEmitter function it's possible for multiple listeners to receive the same notification.

Combine callbacks and EventEmitter

There are also some circumstances where an EventEmitter can be used in conjunction with a callback. This pattern is extremely useful when we want to implement the principle of small surface area by exporting a traditional asynchronous function as the main functionality, while still providing richer features, and more control by returning an EventEmitter. One example of this pattern is offered by the node-glob module (https://npmjs.org/package/glob), a library that performs glob-style file searches. The main entry point of the module is the function it exports, which has the following signature:

glob(pattern, [options], callback)

The function takes pattern as the first argument, a set of options, and a callback function which is invoked with the list of all the files matching the provided pattern. At the same time, the function returns an EventEmitter that provides a more fine-grained report over the state of the process. For example, it is possible to be notified in real-time when a match occurs by listening to the match event, to obtain the list of all the matched files with the end event, or to know whether the process was manually aborted by listening to the abort event. The following code shows how this looks:

var glob = require('glob');
glob('data/*.txt', function(error, files) {
  console.log('All files found: ' + JSON.stringify(files));
}).on('match', function(match) {
  console.log('Match found: ' + match);
});

As we can see, the practice of exposing a simple, clean, and minimal entry point while still providing more advanced or less important features with secondary means is quite common in Node.js, and combining EventEmitter with traditional callbacks is one of the ways to achieve that.

Tip

Pattern: create a function that accepts a callback and returns an EventEmitter, thus providing a simple and clear entry point for the main functionality, while emitting more fine-grained events using the EventEmitter.

 

Summary


In this chapter, we have seen how the Node.js platform is based on a few important principles that provide the foundation to build efficient and reusable code. The philosophy and the design choices behind the platform have, in fact, a strong influence on the structure and behavior of every application and module we create. Often, for a developer moving from another technology, these principles might seem unfamiliar and the usual instinctive reaction is to fight the change by trying to find more familiar patterns inside a world which in reality requires a real shift in the mindset. On one hand, the asynchronous nature of the reactor pattern requires a different programming style made of callbacks and things that happen at a later time, without worrying too much about threads and race conditions. On the other hand, the module pattern and its principles of simplicity and minimalism creates interesting new scenarios in terms of reusability, maintenance, and usability.

Finally, besides the obvious technical advantages of being fast, efficient, and based on JavaScript, Node.js is attracting so much interest because of the principles we have just discovered. For many, grasping the essence of this world feels like returning to the origins, to a more humane way of programming for both size and complexity and that's why developers end up falling in love with Node.js.

In the next chapter, we will focus our attention on the mechanisms to handle asynchronous code, we will see how callbacks can easily become our enemy, and we will learn how to fix that by using some simple principles, patterns, or even constructs that do not require a continuation-passing style programming.

About the Author
  • Mario Casciaro

    Mario Casciaro is a software engineer and entrepreneur, passionate about technology, science and open source knowledge. Mario graduated with a master's degree in software engineering and started his professional career at IBM where he worked for several years on different enterprise products such as Tivoli Endpoint Manager, Cognos Insight, and SalesConnect. Next, he moved to D4H Technologies, a growing SaaS company, to lead the development of a new bleeding-edge product for managing emergency operations in real time. Currently, Mario is the co-founder and CEO of Sponsorama.com, a platform to help online projects raise funds through corporate sponsorship. Mario is also the author of the first edition of Node.js Design Patterns.

    Browse publications by this author
Latest Reviews (4 reviews total)
The material is good. I don't like the epub format reader. it alwasys shows 2 pages wich means the code gets seperated over multiple lines where they should not.That makes the code harder to read.I ended upcopy andpasting most of it into a code editor to get the better view of the code
Livre très instructif avec de nombreux exemples bien choisis. Je le recommande pour tous les niveaux, notamment les dev débutant Node comme moi avec un background front/js et back/php.
It's the best book for Node.js. Thank you, Mario !!!
Node.js Design Patterns
Unlock this book and the full library FREE for 7 days
Start now