Mastering Node.js

Many of the JavaScript extensions in Node emit events. These events are instances of events.EventEmitter. Any object can extend EventEmitter, providing the developer with an elegant toolkit for building tight asynchronous interfaces to object methods.

Work through this example demonstrating how to set an EventEmitter object as the prototype of a function constructor. As each constructed instance now has the EventEmitter object exposed to its prototype chain, this provides a natural reference to the event API (Application Programming Interface). The counter instance methods can therefore emit events, and these can be listened for. Here we emit the latest count whenever the counter.increment method is called, and bind a callback to the incremented event, which simply prints the current counter value to the command line:

var EventEmitter = require('events').EventEmitter;
var Counter = function(init) {
  this.increment = function() {
      init++; 
      this.emit('incremented', init);
  }
}
Counter.prototype = new EventEmitter();
var counter = new Counter(10);
var callback = function(count) {
    console.log(count);
}
counter.addListener('incremented', callback);

counter.increment(); // 11
counter.increment(); // 12

To remove the event listeners bound to counter, use counter.removeListener('incremented', callback). For consistency with browser-based JavaScript, counter.on and counter.addListener are interchangeable.

The addition of EventEmitter as an extensible object greatly increases the possibilities of JavaScript on the server. In particular, it allows I/O data streams to be handled in an event-oriented manner, in keeping with the Node's principle of asynchronous, non-blocking programming:

var Readable = require('stream').Readable;
var readable = new Readable;
var count = 0;

readable._read = function() {
  if(++count > 10) {
    return readable.push(null);
  }
  setTimeout(function() {
    readable.push(count + "\n");
  }, 500);
};
readable.pipe(process.stdout);

In this program we are creating a Readable stream and piping any data pushed into this stream to process.stdout. Every 500 milliseconds we increment a counter and push that number (adding a newline) onto the stream, resulting in an incrementing series of numbers being written to the terminal. When our series has reached its limit (10), we push null onto the stream, causing it to terminate. Don't worry if you don't fully understand how Readable is implemented here—streams will be fully explained in the following chapters. Simply note how the act of pushing data onto a stream causes a corresponding event to fire, how the developer can assign a custom callback to handle this event, and how newly added data can be redirected to other streams. Node is designed such that I/O operations are consistently implemented as asynchronous, evented data streams.

It is also important to note the importance of this style of I/O. Because Node's event loop need only commit resources to handling callbacks, many other instructions can be processed in the down time between each interval.

As an exercise, re-implement the previous code snippet such that the emitted data is piped to a file. You'll need to use fs.createWriteStream:

var fs = require('fs');
var writeStream = fs.createWriteStream("./counter", {
  flags : 'w',
  mode: 0666
});

In his book The Art of Unix Programming, Eric Raymond proposed the Rule of Modularity:

This idea of building complex systems out of small pieces, loosely joined is seen in the management theory, theories of government, physical manufacturing, and many other contexts. In terms of software development, it advises developers to contribute only the simplest and most useful component necessary within a larger system. Large systems are hard to reason about, especially if the boundaries of its components are fuzzy.

One of the primary difficulties when constructing scalable JavaScript programs is the lack of a standard interface for assembling a coherent program out of many smaller ones. For example, a typical web application might load dependencies using a sequence of <script> tags in the <head> section of an HTML document:

<head>
<script src="fileA.js"></script>
<script src="fileB.js"></script>
</head>

There are many problems with this sort of solution:

Ambivalently inserting unpredictable code fragments into an application frustrates attempts to predictably shape functionality. What is needed is a standard way to load and share discreet program modules.

Accordingly, Node introduced the concept of the package, following the CommonJS specification. A package is a collection of program files bundled with a manifest file describing the collection. Dependencies, authorship, purpose, structure, and other important meta-data are exposed in a standard way. This encourages the construction of large systems from many small, interdependent systems. Perhaps even more importantly, it encourages sharing:

In many ways the success of Node is due to growth in the number and quality of packages available to the developer community, distributed via Node's package management system, npm. The design choices of this system, both social and technical, have done much to help make JavaScript a viable professional option for systems programming.

More extensive information on creating and managing Node packages can be found in Appendix A, Organizing Your Work. The key point is this: build programs out of packages where possible, and share those packages when possible. The shape of your applications will be clearer and easier to maintain. Importantly, the efforts of thousands of other developers can be linked into applications via npm, directly by inclusion, and indirectly as shared packages are tested, improved, refactored, and repurposed by members of the Node community.

I/O in the browser is mercilessly hobbled, for very good reasons—if the JavaScript on any given website could access your filesystem, or open up network connections to any server, the WWW would be a less fun place.

For Node, I/O is of fundamental importance, and its focus from the start was to simplify the creation of scalable systems with high I/O requirements. It is likely that your first experience with Node was in writing an HTTP server.

Node supports several standard network protocols in addition to HTTP, such as TLS/SSL (Transport Layer Security/Secure Sockets Layer), and UDP (User Datagram Protocol). With these tools we can easily build scalable network programs, moving well beyond the somewhat dated AJAX (Asynchronous JavaScript And Xml) techniques familiar to the JavaScript developer.

Let's create a simple program that allows the user to send data between two UDP servers:

var dgram = require('dgram');
var client = dgram.createSocket("udp4");
var server = dgram.createSocket("udp4");

var message = process.argv[2] || "message";

message = new Buffer(message);

server
.on("message", function (msg) {
  process.stdout.write("Got message: " + msg + "\n");
  process.exit();
  
})
.bind(41234);

client.send(message, 0, message.length, 41234, "localhost");

Assuming a program file name of udp.js a message can be sent via UDP by running this program from the terminal like so:

node udp.js "my message"

Which will result in the following output:

Got message: my message

We first establish our UDP servers, one working as a broadcaster, the other as a listener. process.argv contains useful command information, including command-line arguments commencing at index(2), which in this case would contain "my message". UDP requires messages to be Buffer objects, so we ensure that some message exists and convert it.

A UDP server is an instance of EventEmitter, emitting a message event when messages are received on the port it is bound. This server simply echoes the received message. All that is left to do is send the message, which action is performed by the client, passing along our message to port #41234.

Moving streams of data around the I/O layer of your application is simplified within Node. It isn't difficult to share data streams across differing protocol servers, as data streams are standardized via Node's interfaces. Protocol details are handled for you.

Let's continue to explore I/O, the process object, and events. First, let's dig into the machine powering Node's core.

One very powerful V8 configuration option is important enough to make it into Node's own collection: --max-stack-size. Let's look into some of the new powers a Node developer has been given in being able to configure a specific JavaScript runtime.

Trying to break a system is an excellent way to discover its limits and shape. Let's write a program that will crash V8:

var count = 0;
(function curse() { 
  console.log(++count);
  curse();
})()

This self-contained, self-executing function will recursively call itself forever, or until it is forced to stop. Each iteration of curse adds another frame to the call stack. This uncontrolled growth will eventually cause the JavaScript runtime to collapse, citing a RangeError: Maximum call stack size exceeded.

The purpose of --max-stack-size should now be clear. The direct V8 option equivalent is –-stack_size, which is passed a value, in KB (Kilobytes), to raise this limit. Experiment with the above program, noting the number of iterations possible at different settings.

While it is likely that hitting this limit represents an incorrectly designed algorithm, being able to expand the width of the operating space available to a Node process adds to the collection of solutions available to developers.

On 32 bit and 64 bit machines V8's memory allocation defaults are, respectively, 700 MB and 1400 MB. In newer versions of V8, memory limits on 64 bit systems are no longer set by V8, theoretically indicating no limit. However, the OS (Operating System) on which Node is running can always limit the amount of memory V8 can take, so the true limit of any given process cannot be generally stated.

V8 makes available the --max_old_space_size option, which allows control over the amount of memory available to a process, accepting a value in MB. Should you need to increase memory allocation, simply pass this option the desired value when spawning a Node process.

It is often an excellent strategy to reduce the available memory allocation for a given Node instance, especially when running many instances. As with stack limits, consider whether massive memory needs are better delegated to a dedicated storage layer, such as an in-memory database or similar.

One of the key advantages of modern high-level languages such as JavaScript is the automatic management of memory through GC (Garbage Collection). GC strategies are many and complex, yet all follow a simple core idea: every so often, free allocated memory that is no longer being used.

The drawback of automatic GC is that it puts a slight brake on process speed. While the clear advantages of automatic GC outweigh its drawbacks in the majority of cases, there remains for the Node developer an opportunity to control some of its behavior. This is primarily done via the flags –-nouse_idle_notification and –-expose_gc.

Passing the –-nouse_idle_notification flag will tell V8 to ignore idle notification calls from Node, which are requests to V8 asking it to run GC immediately, as the Node process is currently idle. Because Node is aggressive with these calls (efficiency breeds clean slates), an excess of GC may slow down your application. Note that using this flag does not disable GC; GC simply runs less often. In the right circumstances this technique can increase performance.

--expose_gc introduces a new global method to the Node process, gc(), which allows JavaScript code to manually start the GC process. In conjunction with –-nouse_idle_notification the developer can now control to some degree how often GC runs. At any point in my JavaScript code I can simply call gc() and start the collector.

Being able to adjust memory usage and GC is certainly useful. Remember that application volume can rapidly rise in unpredictable ways. If the memory footprint of your application holds steady near the very limits of V8's allocation, you should begin to think about scaling horizontally. Use memory wisely, and split off new Node instances where appropriate.

JavaScript has never stopped evolving and it is now experiencing something of a renaissance, helped in no small part by the popularity of Node. The language's next version, named Harmony, introduces some significant new features and concepts.

The available Harmony options are:

It is beyond the scope of this book to discuss these new features in any depth. Nonetheless, it should be stated that this ability to use the very latest JavaScript features, now, through Node, offers the developer a great advantage—there is no browser war on a server to hold back innovation.

For example, ES6's weakMap allows the use of non-strings as keys in a HashMap:

"use strict"
let owners = new WeakMap();
let task = {
  title 	: "Big Project"
};
owners.set(task, 'John');
function owner(task) {
  if(owners.has(task)) {
    return console.log(owners.get(task));
  }
  console.log("No owner for this task.");
}
owner(task);   // "John"
owner({});   // "No owner for this task"

As an exercise, the reader might map (fixed) input streams to (variable) output streams in a similar manner.