About this book

Using the same framework to build both server and client-side applications saves you time and money. This book teaches you how you can use JavaScript and Node.js to build highly scalable APIs that work well with lightweight cross-platform client applications. It begins with the basics of Node.js in the context of backend development, and quickly leads you through the creation of an example client that pairs up with a fully authenticated API implementation. By the end of the book, you’ll have the skills and exposure required to get hands-on with your own API development project.

Publication date:
July 2018
Publisher
Packt
Pages
100
ISBN
9781789539660

 

Chapter 1. Introduction to Node.js

This chapter is designed to cover a few fundamental concepts in Node.js, as we lay a foundation for our subsequent chapters on API development.

Let's start this first chapter with a quick dive into how Node.js works and where it's being used lately. We will then have a look at its module system and its asynchronous programming model. Let's get started.

By the end of this chapter, you will be able to:

  • Describe the basics of how Node.js works
  • List the applications of Node.js in modern software development
  • Describe the module system used by Node.js
  • Implement basic modules for an application
  • Explain the asynchronous programming basics in Node.js
  • Implement a basic application using async/await
 

The Basics of Node.js


Node.js is an event-driven, server-side JavaScript environment. Node.js runs JS using the V8 engine developed by Google for use in their Chrome web browser. Leveraging V8 allows Node.js to provide a server-side runtime environment that compiles and executes JS at lightning speeds.

Node.js runs as a single-threaded process that acts upon callbacks and never blocks on the main thread, making it high-performing for web applications. A callback is basically a function that is passed to another function so that it can be called once that function is done. We will look into this in a later topic. This is known as the single-threaded event loop model. Other web technologies mainly follow the multithreaded request-response architecture.

The following diagram depicts the architecture of Node.js. As you can see, it's mostly C++ wrapped by a JavaScript layer. We will not go over the details of each component, since that is out of the scope of this chapter.

Node's goal is to offer an easy and safe way to build high-performance and scalable network applications in JavaScript.

Applications of Node.js

Node.js has the following four major applications:

  • Creating REST APIs: We are going to look into this more in subsequent chapters
  • Creating real-time services: Because of Node's asynchronous event-driven programming, it is well-suited to reactive real-time services
  • Building microservices: Since Node.js has a very lean core, it is best suited to building microservices, since you will only add dependencies that you actually need for the microservices, as opposed to the glut that comes with other frameworks
  • Tooling: For example, DevOps automations, and so on

Activity: Running Basic Node.js Code

Before You Begin

Open the IDE and the Terminal to implement this solution.

Aim

Learn how to write a basic Node.js file and run it.

Scenario

You are writing a very basic mathematical library with handy mathematical functions.

Steps for Completion

  1. Create your project directory (folder), where all the code for this and other chapters will be kept. You can call it beginning-nodejs for brevity. Inside this directory, create another directory named lesson-1, and inside that, create another directory called activity-a. All this can be done using the following command:
mkdir -p beginning-nodejs/lesson-1/activity-a
  1. Inside activity-a, create a file using touch maths.js command.  
  2. Inside this file, create the following functions:
    • add: This takes any two numbers and returns the sum of both, for example, add(2, 5) returns 7
    • sum: Unlike add, takes any number of numbers and returns their sum, for example, sum(10, 5, 6) returns 21
  3.  After these functions, write the following code to act as tests for your code:
console.log(add(10, 6)); // 16
console.log(sum(10, 5, 6)); // 21
  1. Now, on the Terminal, change directory to lesson-1. That's where we will be running most of our code from for the whole chapter.
  2. To run the code, run the following command:
node activity-a/math.js

The 16 and 21 values should be printed out on the Terminal.

Note

Even though you can configure the IDE so that Node.js code be run at the click of a button, it's strongly recommend that you run the code from the Terminal to appreciate how Node.js actually works. For uniformity, if you are using a Windows machine, then run your commands from the Git Bash Terminal. For the reference solution, use the math.js file at Code/Lesson-1/activity-solutions/activity-a.

 

The Module System


Let's have a look at Node's module system and the different categories of the Node.js modules.

Application Modularization

Like most programming languages, Node.js uses modules as a way of organizing code. The module system allows you to organize your code, hide information, and only expose the public interface of a component using module.exports.

Node.js uses the CommonJS specification for its module system:

  • Each file is its own module, for instance, in the following example, index.js and math.js are both modules
  • Each file has access to the current module definition using the module variable
  • The export of the current module is determined by the module.exports variable
  • To import a module, use the globally available require function

Let's look at a simple example:

// math.js file
function add(a, b) 
{
  return a + b;
}
…
…
module.exports = 
{
  add,
  mul,
  div,
};
// index.js file
const math = require('./math');
console.log(math.add(30, 20)); // 50

Note

To call other functions such as mul and div, we'll use object destructuring as an alternative when requiring the module, for example, const { add } = require('./math');. The code files for the section The Module System are placed at Code/Lesson-1/b-module-system.

Module Categories

We can place Node.js modules into three categories:

  • Built-in (native) modules: These are modules that come with Node.js itself; you don't have to install them separately.
  • Third-party modules: These are modules that are often installed from a package repository. npm is a commonly used package repository, but you can still host packages on GitHub, your own private server, and so on.
  • Local modules: These are modules that you have created within your application, like the example given previously.

Built-In Modules

As mentioned earlier, these are modules that can be used straight-away without any further installation. All you need to do is to require them. There are quite a lot of them, but we will highlight a few that you are likely to come across when building web applications:

  • assert: Provides a set of assertion tests to be used during unit testing
  • buffer: To handle binary data
  • child_process: To run a child process
  • crypto: To handle OpenSSL cryptographic functions
  • dns: To do DNS lookups and name resolution functions
  • events: To handle events
  • fs: To handle the filesystem
  • http or https: For creating HTTP(s) servers
  • stream: To handle streaming data
  • util: To access utility functions like deprecate (for marking functions as deprecated), format (for string formatting), inspect (for object debugging), and so on

For example, the following code reads the content of the lesson-1/temp/sample.txt file using the in-built fs module:

const fs = require('fs');
let file = `${__dirname}/temp/sample.txt`;
fs.readFile(file, 'utf8', (err, data) => 
{
  if (err) throw err;
  console.log(data);
});

The details of this code will be explained when we look at asynchronous programming later in this chapter.

npm – Third-Party Module Registry

Node Package Manager (npm) is the package manager for JavaScript and the world's largest software registry, enabling developers to discover packages of reusable code.

To install an npm package, you only need to run the command npm install <package-name> within your project directory. We are going to use this a lot in the next two chapters.

Let's look at a simple example. If we wanted to use a package (library) like request in our project, we could run the following command on our Terminal, within our project directory:

npm install request

To use it in our code, we require it, like any other module:

const request = require('request');
request('http://www.example.com', (error, response, body) => 
{
  if (error) console.log('error:', error); // Print the error if one occurred
  else console.log('body:', body); // Print the HTML for the site.
});

Note

More details about npm can be found here: https://docs.npmjs.com/. Recently, a new package manager was released called YARN (https://docs.npmjs.com/), which is becoming increasingly popular. When you run the npm install <module-name> command on your project for the first time, the node_modules folder gets created at the root of your project.

Scanning for node_modules

It's worth noting how Node.js goes about resolving a particular required module. For example, if a file /home/tony/projects/foo.js has a require call require('bar'), Node.js scans the filesystem for node_modules in the following order. The first bar.js that is found is returned:

  • /home/tony/projects/node_modules/bar.js
  • /home/tony/node_modules/bar.js
  • /home/node_module/bar.js
  • /node_modules/bar.js

Node.js looks for node_moduels/bar in the current folder followed by every parent folder until it reaches the root of the filesystem tree for the current file.

Note

The module foo/index.js can be required as foo, without specifying index, and will be picked by default.

Handy npm Commands

Let's dive a little deeper into npm, by looking at some of the handy npm commands that you will often use:

  • npm init: Initializes a Node.js project. This should be run at the root of your project and will create a respective package.json file. This file usually has the following parts (keys):
    • name: Name of the project.
    • version: Version of the project.
    • description: Project description.
    • main: The entry-point to your project, the main file.
    • scripts: This will be a list of other keys whose values will be the scripts to be run, for example, test, dev-server. Therefore, to run this script, you will only need to type commands such as npm run dev-server, npm run test, and so on.
    • dependencies: List of third-party packages and their versions used by the project. Whenever you do npm install <package-name> --save, this list is automatically updated.
    • devDependencies: List of third-party packages that are not required for production, but only during development. This will usually include packages that help to automate your development workflow, for example, task runners like gulp.js. This list is automatically updated whenever you do npm install <package-name> --save-dev.
    • npm install: This will install all the packages, as specified in the package.json file.
  • npm install <package-name> <options>:
    • With the --save option, installs the package and saves the details in the package.json file.
    • With the --save-dev option, installs the package and saves the details in the package.json, under devDependencies.
    • With the --global option, installs the package globally in the whole system, not only in the current system. Due to permissions, this might require running the command with administrator rights, for example, sudo npm install <package-name> --global.
    • npm install <package-name>@<version>, installs a specific version of a package. Usually, if a version is not specified, the latest version will be installed.
  • npm list: Lists the packages that have been installed for the project, reading from what is installed in node_modules.
  • npm uninstall <package-name>: Removes an installed package.
  • npm outdated: Lists installed packages that are outdated, that is, newer versions have been released.

Local Modules

We have already looked at how local modules are loaded from the previous example that had math.js and index.js.

Since JavaScript Object Notation (JSON) is such an important part of the web, Node.js has fully embraced it as a data format, even locally. You can load a JSON object from the local filesystem the same way you load a JavaScript module. During the module loading sequence, whenever a file.js is not found, Node.js looks for a file.json.

See the example files in lesson-1/b-module-system/1-basics/load-json.js:

const config = require('./config/sample');
console.log(config.foo); // bar

Here, you will notice that once required, the JSON file is transformed into a JavaScript object implicitly. Other languages will have you read the file and perhaps use a different mechanism to convert the content into a data structure such as a map, a dictionary, and so on.

Note

For local files, the extension is optional, but should there be a conflict, it might be necessary to specify the extension. For example, if we have both a sample.js and a sample.json file in the same folder, the .js file will be picked by default; it would be prudent to specify the extension, for example: const config = require('./config/sample.json'); When you run npm install, without specifying the module to install, npm will install the list of packages specified (under dependencies and devDependencies in the package.json file in your project). If package.json does not exist, it will give an error indicating that no such file has been found.

Activity: Using a Third-Party Package for the Previous math.js Code

Before You Begin

This activity will build upon the, Running Basic Node.js activity of this chapter.

Aim

If the argument is a single array, sum up the numbers, and if it's more than one array, first combine the arrays into one before summing up. We will use the concat() function from lodash, which is a third-party package that we will install.

Scenario

We want to create a new function, sumArray, which can sum up numbers from one or more arrays.

Steps for Completion

  1. Inside Lesson-1, create another folder called activity-b.
  2. On the Terminal, change directory to activity-b and run the following command:
npm init
  1. This will take you to an interactive prompt; just press Enter all the way, leaving the answers as suggested defaults. The aim here is for us to get a package.json file, which will help us organize our installed packages.
  2. Since we will be using lodash, let's install it. Run the following command:
npm install lodash--save

Notice that we are adding the --save option on our command so that the package installed can be tracked in package.json. When you open the package.json file created in step 3, you will see an added dependencies key with the details.

  1. Create a math.js file in the activity-b directory and copy the math.js code from Activity, Running Basic Node.js into this file.
  2. Now, add the sumArray function right after the sum function.
  3. Start with requiring lodash, which we installed in step 4, since we are going to use it in the sumArray function:
const _ = require('lodash');
  1. The sumArray function should call the sum function to reuse our code. Hint: use the spread operator on the array. See the following code:
function sumArray() 
{
  let arr = arguments[0];
  if (arguments.length > 1) 
  {
    arr = _.concat(...arguments);
  }
  // reusing the sum function
  // using the spread operator (...) since
  // sum takes an argument of numbers
  return sum(...arr);
}
  1. At the end of the file, export the three functions, add, sum, and sumArray with module.exports.
  2. In the same activity-b folder, create a file, index.js.
  3. In index.js file, require ./math.js and go ahead to use sumArray:
// testing
console.log(math.sumArray([10, 5, 6])); // 21
console.log(math.sumArray([10, 5], [5, 6], [1, 3])) // 30
  1. Run the following code on the Terminal:
node index.js

You should see 21 and 30 printed out.

Note

The solution files are placed at Code/Lesson-1/activitysolutions/activity-b.

 

Asynchronous Programming with Node.js


Let's have a look at asynchronous programming model that is at the heart of how Node.js works.

Callbacks

Callbacks are functions that are executed asynchronously, or at a later time. Instead of the code reading top to bottom procedurally, asynchronous programs may execute different functions at different times based on the order and speed of earlier functions.

Since JavaScript treats functions like any other object, we can pass a function as an argument in another function and alter execute that passed-in function or even return it to be executed later.

We saw such a function previously when we were looking at the fs module in The Module System section. Let's revisit it:

const fs = require('fs');
let file = `${__dirname}/temp/sample.txt`;
fs.readFile(file, 'utf8', (err, data) => 
{
  if (err) throw err;
  console.log(data);
});

Note

The code files for Asynchronous Programming with Node.js are placed at Code/Lesson-1/c-async-programming.

On line 3, we use a variable part of the globals, _ _dirname, which basically gives us the absolute path of the directory (folder) in which our current file (read-file.js) is, from which we can access the temp/sample.txt file.

Our main point of discussion is the chunk of code between lines 5 and 8. Just like most of the methods you will come across in Node.js, they mostly take in a callback function as the last argument.

Most callback functions will take in two parameters, the first being the error object and the second, the results. For the preceding case, if file reading is successful, the error object, err, will be null and the contents of the file will be returned in the data object.

Let's break down this code for it to make more sense:

const fs = require('fs');
let file = `${__dirname}/temp/sample.txt`;
const callback = (err, data) => 
{
  if (err) throw err;
  console.log(data);
};
fs.readFile(file, 'utf8', callback);

Now, let's look at the asynchronous part. Let's add an extra line to the preceding code:

const fs = require('fs');
let file = `${__dirname}/temp/sample.txt`;
const callback = (err, data) => 
{
  if (err) throw err;
  console.log(data);
};
fs.readFile(file, 'utf8', callback);
console.log('Print out last!');

See what we get as a print out:

Print out last!
  hello,
  world

How come Print out last! comes first? This is the whole essence of asynchronous programming. Node.js still runs on a single thread, line 10 executes in a non-blocking manner and moves on to the next line, which is console.log('Print out last!'). Since the previous line takes a long time, the next one will print first. Once the readFile process is done, it then prints out the content of file through the callback.

Promises

Promises are an alternative to callbacks for delivering the results of an asynchronous computation. First, let's look at the basic structure of promises, before we briefly look at the advantages of using promises over normal callbacks.

Let's rewrite the code above with promises:

const fs = require('fs');
const readFile = (file) => 
{
  return new Promise((resolve, reject) => 
  {
    fs.readFile(file, 'utf8', (err, data) => 
    {
      if (err) reject(err);
      else resolve(data);
    });
  });
}
// call the async function
readFile(`${__dirname}/../temp/sample.txt`)
  .then(data => console.log(data))
  .catch(error => console.log('err: ', error.message));

This code can further be simplified by using the util.promisify function, which takes a function following the common Node.js callback style, that is, taking an (err, value) => … callback as the last argument and returning a version that returns promises:

const fs = require('fs');
const util = require('util');
const readFile = util.promisify(fs.readFile);
readFile(`${__dirname}/../temp/sample.txt`, 'utf8')
  .then(data => console.log(data))
  .catch(error => console.log('err: ', error));

From what we have seen so far, promises provide a standard way of handling asynchronous code, making it a little more readable.

What if you had 10 files, and you wanted to read all of them? Promise.all comes to the rescue. Promise.all is a handy function that enables you to run asynchronous functions in parallel. Its input is an array of promises; its output is a single promise that is fulfilled with an array of the results:

const fs = require('fs');
const util = require('util');
const readFile = util.promisify(fs.readFile);
const files = [
  'temp/sample.txt',
  'temp/sample1.txt',
  'temp/sample2.txt',
];
// map the files to the readFile function, creating an
// array of promises
const promises = files.map(file => readFile(`${__dirname}/../${file}`, 'utf8'));
Promise.all(promises)
  .then(data => 
  {
    data.forEach(text => console.log(text));
  })
  .catch(error => console.log('err: ', error));

Async/Await

This is one of the latest additions to Node.js, having been added early in 2017 with version 7.6, providing an even better way of writing asynchronous code, making it look and behave a little more like synchronous code.

Going back to our file reading example, say you wanted to get the contents of two files and concatenate them in order. This is how you can achieve that with async/await:

const fs = require('fs');
const util = require('util');
const readFile = util.promisify(fs.readFile);
async function readFiles() 
{
  const content1 = await readFile(`${__dirname}/../temp/sample1.txt`);
  const content2 = await readFile(`${__dirname}/../temp/sample2.txt`);
  return content1 + '\n - and - \n\n' + content2;
}
readFiles().then(result => console.log(result));

In summary, any asynchronous function that returns a promise can be awaited.

Activity: Transforming a Text File Using an Async Function

Before You Begin

You should have already gone through the previous activities.

Aim

Read the file (using fs.readFile), in-file.txt, properly case format the names (using the lodash function, startCase), then sort the names in alphabetical order and write them out to a separate file out-file.txt (using fs.writeFile).

Scenario

We have a file, in-file.txt, containing a list of peoples' names. Some of the names have not been properly case formatted, for example, john doe should be changed to John Doe.

Steps for Completion

  1. In Lesson-1, create another folder called activity-c.

  2. On the Terminal, change directory to activity-c and run the following command:
npm init
  1. Just like in the previous activity, this will take you to an interactive prompt; just press Enter all the way, leaving the answers as suggested defaults. The aim here is for us to get a package.json file, which will help us organize our installed packages.
  2. Since we will be using lodash here too, let's install it. Run, npm install lodash --save.
  3. Copy the in-file.txt file provided in the student-files directory into your activity-c directory.
  4. In your activity-c directory, create a file called index.js, where you will write your code.
  5. Now, go ahead and implement an async function transformFile, which will take the path to a file as an argument, transform it as described previously (under Aim), and write the output to an output file provided as a second parameter.
  6. On the Terminal, you should indicate when you are reading, writing, and done, for example:
    • reading file: in-file.txt
    • writing file: out-file.txt
    • done

Note

You should read the quick reference documentation on fs.writeFile since we haven't used it yet. However, you should be able to see its similarity with fs.readFile, and convert it into a promise function, as we did previously.

The solution files are placed at Code/Lesson-1/activitysolutions/activity-c.

 

Summary


In this chapter, we went through a quick overview of Node.js, seeing how it looks under the hood.

We wrote basic Node.js code and ran it from the Terminal using the Node.js command.

We also looked at module system of Node.js, where we learnt about the three categories of Node.js modules, that is, in-built, third-party (installed from the npm registry), and local modules, and their examples. We also looked at how Node.js resolves a module name whenever you require it, by searching in the various directories.

We then finished off by looking at the asynchronous programming model that is at the heart of how Node.js works, and what actually makes Node.js tick. We looked at the three main ways you can write asynchronous code: using callbacks, Promises, and the new async/await paradigm.

The foundation is now laid for us to go ahead and implement our API using Node.js. Most of these concepts will crop up again as we build our API.

About the Author

  • Anthony Nandaa

    Anthony Nandaa is a senior software developer with more than 7 years of professional programming experience. He was introduced to programming 3 years before his career as a developer began, working with Pascal and Visual Basic 6. In his career so far, he has worked with multiple languages, such as Python, PHP, Go, and full-stack JavaScript. In his current role, he leads a team of engineers working with Node.js and React for frontend development. He considers himself a lifelong learner, and lately, he has been learning Haskell for fun and to gain some insight into pure functional programming.

    Browse publications by this author

Latest Reviews

(1 reviews total)
Some of the codes in the book are wrong i bought. The SQL query especially.

Recommended For You

Node.js Complete Reference Guide

Node.js Complete Reference Guide is your one-stop solution to building highly scalable, single-language applications that share code between the server and client

By Valentin Bojinov and 2 more
Advanced JavaScript

Gain a deeper understanding of JavaScript and apply it to build small applications in backend, frontend, and mobile frameworks.

By Zachary Shute
Python API Development Fundamentals

Learn all that’s needed to build a fully functional web application from scratch.

By Jack Chan and 2 more
Learn React Hooks

Create large-scale web applications with code that is extensible and easy to understand using React Hooks

By Daniel Bugl