For years, being a frontend developer meant writing a bit of HTML and putting some styling with CSS on it. However, since the last decade, this job description barely holds true. In contrast, the majority of frontend work is now done using JavaScript.

Initially used to make cosmetic enhancements to websites (such as the toggling of elements) possible, frontend development is now the glue of the web. Websites are no longer just written in HTML and CSS. Instead, in many cases, web pages are programmed with JavaScript using modern techniques such as dependency management and bundling of resources. The Node.js framework provides an ideal foundation for this movement. It enables developers to use JavaScript not only inside websites running in a browser but also within the tooling to write web pages – outside of a browser.

When Node.js was released in May 2009, it did not seem like a big deal. JavaScript was working on the server too. However, the cross-platform nature of Node.js and the size of the JavaScript community provided the basis for one of the greatest disruptions in the history of computing. People started adopting the framework so quickly that many existing frameworks either disappeared or had to be reworked to stay attractive to developers. Soon, JavaScript was used in the browser and on the server and was also part of every frontend developer’s toolbox.

With the rise of new development frameworks such as Angular or React, the need for attractive frontend tooling became apparent. The new frameworks always relied on some build steps – otherwise, websites and applications using these frameworks would have been far too inconvenient to write for developers. Since the vast Node.js ecosystem seemed to have figured out a suitable approach for reusability, these new frameworks adopted it and made it an integral part of their development story. This way, using Node.js became the de facto standard for frontend projects of any kind.

Today, it is pretty much impossible to start a frontend development project without having Node.js installed. In this book, we’ll take the journey of learning about Node.js from the inside out together. We will not be focusing on writing server applications or walking over the integrated functionality of Node.js. Instead, we’ll look at how we – as frontend developers – can leverage the best that Node.js brings to the table.

In this first chapter, we discuss the internals of Node.js. This will help you understand how Node.js works and how you can actually use it. After this chapter, you will be able to run and debug simple scripts using the Node.js command-line application.

We will cover the following key topics in this chapter:

• Looking at the Node.js architecture in detail
• Understanding the event loop
• Using Node.js from the command line
• CommonJS

To follow the code samples in this book, you need knowledge of JavaScript and how to use the command line. You should have Node.js installed using the instructions at https://nodejs.org.

The complete source code for this chapter is available at https://github.com/PacktPublishing/Modern-Frontend-Development-with-Node.js/tree/main/Chapter01.

The Code in Action (CiA) videos for this chapter can be accessed at http://bit.ly/3fPPdtb.

The principal foundations of Node.js have been inspired by a few things:

• The single worker thread featured in browsers was already quite successful in the server space. Here, the popular nginx web server showed that the event loop pattern (explained later in this chapter) was actually a blessing for performance – eliminating the need to use a dedicated thread pool for handling requests.
• The idea of packaging everything in a file-centric structure called modules. This allowed Node.js to avoid many of the pitfalls of other languages and frameworks – including JavaScript in the browser.
• The idea of avoiding creating a huge framework and leaving everything extensible and easy to get via package managers.

Threads

Modern computers offer a lot of computing power. However, for an application to really use the available computing power, we need to have multiple things working in parallel. Modern operating systems know about different independently running tasks via so-called threads. A thread is a group of operations running sequentially, which means in a given order. The operating system then schedules when threads run and where (i.e., on which CPU core) they are placed.

These principles together form a platform that seems easy to create, but hard to replicate. After all, there are plenty of JavaScript engines and useful libraries available. For Ryan Dahl, the original creator and maintainer of Node.js, the basis of the framework had to be rock solid.

Ryan Dahl selected an existing JavaScript engine (V8) to take over the responsibility of parsing and running the code written in JavaScript. The V8 engine was chosen for two good reasons. On the one hand, the engine was available as an open source project under a permissive license – usable by projects such as Node.js. On the other hand, V8 was also the engine used by Google for its web browser Chrome. It is very fast, very reliable, and under active development.

One of the drawbacks of using V8 is that it was written in C++ using custom-built tooling called GYP. While GYP was replaced in V8 years later, the transition was not so easy for Node.js. As of today, Node.js is still relying on GYP as a build system. The fact that V8 is written in C++ seems like a side note at first, but might be pretty important if you ever intend to write so-called native modules.

Native modules allow you to go beyond JavaScript and Node.js – making full use of the available hardware and system capabilities. One drawback of native modules is that they must be built on each platform. This is against the cross-platform nature of Node.js.

Let’s take a step back to arrange the parts mentioned so far in an architecture diagram. Figure 1.1 shows how Node.js is composed internally:

Figure 1.1 – Internal composition of Node.js

The most important component in Node.js’s architecture – besides the JavaScript engine – is the libuv library. libuv is a multi-platform, low-level library that provides support for asynchronous input/output (I/O) based on an event loop. I/O happens in multiple forms, such as writing files or handling HTTP requests. In general, I/O refers to anything that is handled in a dedicated area of the operating system.

Any application running Node.js is written in JavaScript or some flavor of it. When Node.js starts running the application, the JavaScript is parsed and evaluated by V8. All the standard objects, such as console, expose some bindings that are part of the Node.js API. These low-level functions (such as console.log or fetch) make use of libuv. Therefore, some simple script that only works against language features such as primitive calculations (2 + 3) does not require anything from the Node API and will remain independent of libuv. In contrast, once a low-level function (for example, a function to access the network) is used, libuv can be the workforce behind it.

In Figure 1.2, a block diagram illustrating the various API layers is shown. The beauty of this diagram is that it reveals what Node.js actually is: a JavaScript runtime allowing access to low-level functionality from state-of-the-art C/C++ libraries. The Node.js API consists of the included Node.js bindings and some C/C++ addons:

Figure 1.2 – Composition of Node.js in terms of building blocks

One thing that would need explanation in the preceding diagram is how the event loop is implemented in relation to all the blocks. When talking about Node.js’s internal architecture, a broader discussion of what an event loop is and why it matters for Node.js is definitely required. So let’s get into these details.

An event loop is a runtime model that enables users to run all operations from a single thread – irrespective of whether the operations access long-running external resources or not. For this to work, the event loop needs to make requests to an event provider, which calls the specified event handlers. In Node.js, the libuv library is used for event loop implementation.

The reason for giving libuv the most space in Figure 1.1 is to highlight the importance of this library. Internally, libuv is used for everything regarding I/O, which arguably is the most crucial piece of any framework. I/O lets a framework communicate with other resources, such as files, servers, or databases. By default, dealing with I/O is done in a blocking manner. This means that the sequence of operations in our application is essentially stopped, waiting for the I/O operation to finish.

Two strategies for mitigating the performance implications of blocking I/O exist.

The first strategy is to create new threads for actually performing these blocking I/O operations. Since a thread contains an independent group of operations, it can run concurrently, eventually not stopping the operations running in the original thread of the application.

The second strategy is to not use blocking I/O at all. Instead, use an alternative variant, which is usually called non-blocking I/O or asynchronous I/O. Non-blocking I/O works with callbacks, that is, functions that are called under certain conditions – for instance when the I/O operation is finished. Node.js uses libuv to make extensive use of this second strategy. This allows Node.js to run all code in a single thread, while I/O operations run concurrently.

In Figure 1.3, the building blocks of libuv are displayed. The key part is that libuv already comes with a lot of functionality to handle network I/O. Furthermore, file and DNS operations are also covered well:

Figure 1.3 – Building blocks of libuv

In addition to the different I/O operations, the library comes with a set of different options for handling asynchronous user code.

The event loop itself follows the reactor design pattern. Wikipedia describes the pattern as follows:

The reactor design pattern is an event handling pattern for handling service requests delivered concurrently to a service handler by one or more inputs. The service handler then demultiplexes the incoming requests and dispatches them synchronously to the associated request handlers. (https://en.wikipedia.org/wiki/Reactor_pattern)

Importantly, this definition mentions synchronous dispatch. This means that code that is run through the event loop is guaranteed to not run into any conflicts. The event loop makes sure that code is always run sequentially. Even though the I/O operations may concurrently run, our callbacks will never be invoked in parallel. From our perspective, even though Node.js will internally (through libuv) use multiple threads, the whole application is single-threaded.

The following is a simple script that shows you the basic behavior of the event loop at play – we’ll discuss how to run this in the Using Node.js from the command line section:

events.js

console.log('A [start]');
setTimeout(() => console.log('B [timeout]'), 0);
Promise.resolve().then(() => console.log('C [promise]'));
console.log('D [end]');

We will run this script in the next section when we learn about the command line usage of Node.js. In the meantime, put some thought into the preceding code and write down the order in which you’ll see the console output. Do you think it will print in an “A B C D” order, or something else?

The algorithm of the implementation of the event loop in libuv is displayed in Figure 1.4:

Figure 1.4 – The implementation of the event loop in libuv

While the code snippet only deals with JavaScript-related constructs (such as console, Promise, and setTimeout), in general, the callbacks are associated with resources that go beyond Node.js, such as file system changes or network requests. Some of these resources may have an operating system equivalent; others only exist in form of blocking I/O.

Consequently, the event loop implementation always considers its thread pool and polls for progressed I/O operations. Timers (such as setTimeout in the example script) are only run in the beginning. To know whether a timer needs to be run, its due time is compared with the current time. The current time is synced with the system time initially. If there is nothing to be done anymore (that is, no active timer, no resource waiting to finish, etc.), then the loop exits.

Let’s see how we can run Node.js to solidify our knowledge about the event loop.

Using JavaScript for a web application just requires you to open the website in a browser. The browser will evaluate the included JavaScript and run it. When you want to use JavaScript as a scripting language, you need to find a new way of running JavaScript. Node.js offers this new way – running JavaScript in a terminal, inside our computer, or from a server.

When Node.js is installed, it comes with a set of command-line tools that will be available in the terminal of your choice. For this book, you’ll need to know about three different executables that we’ll use throughout the chapters:

• node: The main application to run a Node.js script
• npm: The default package manager – more on that later
• npx: A very convenient utility to run npm binaries

For now, we only need to know about node. If we want to run the events.js script from the previous section, we need to execute the following command in the directory in which the script (events.js) has been placed. You can place it there by just inserting the content from the previous events.js listing:

$ node events.js

A [start]

D [end]

C [promise]

B [timeout]

The command is shown after the conventional $ sign indicating the command prompt. The output of running the script is shown below the node events.js command.

As you can see, the order is “A D C B” – that is, Node.js first handled all the sequential operations before the callbacks of the promise were handled. Finally, the timeout callback was handled.

The reason for handling the promise callback before the timeout callback lies in the event loop. In JavaScript, promises spawn so-called micro tasks, which are placed in the pending callback section of the libuv event loop from Figure 1.4. The timeout callback, however, is treated like a full task. The difference between them lies within the event loop. Micro tasks are placed in an optimized queue that is actually peeked multiple times per event loop iteration.

According to libuv, the timeout callback can only be run when its timer is due. Since we only placed it in the event loop during the idle handles (i.e., main section) of the event loop, we need to wait until the next iteration of the event loop.

The node command-line application can also receive additional parameters. The official documentation goes into all details (https://nodejs.org/api/cli.html). A helpful one is -e (short version of --eval) to just evaluate a script directly from the command-line input without requiring a file to run:

$ node -e "console.log(new Date())"

2022-04-29T09:20:44.401

Another very helpful command line flag is --inspect. This opens the standard port for graphical inspection, for example, via the Chrome web browser.

Let’s run an application with a bit of continuous logic to justify an inspection session. In the terminal on your machine, run the following:

$ node -e "setInterval(() => console.log(Math.random()), 60 * 1000)" --inspect

Debugger listening on ws://127.0.0.1:9229/64c26b8a-0ba9-484f-902d-759135ad76a2

For help, see: https://nodejs.org/en/docs/inspector

Now we can run a graphical application. Let’s use the Chrome web browser. Open it and go to chrome://inspect. This is a special Chrome-internal URL that allows us to see the available targets.

The following figure (Figure 1.5) shows how inspecting the Node.js application in the Chrome web browser may look:

Figure 1.5 – Inspecting the Node.js application in the Chrome web browser

In this case, Chrome detected our application with the process ID 3420 running. On your machine, the process ID will most likely be different. No filename was given, as we started with the -e command-line option.

When you click on inspect, you’ll open the standard Chrome DevTools. Instead of debugging a website, you can now debug the Node.js application. For instance, you’ll already get the same console output that you see in the command line.

When you follow the link to the evaluated script from the DevTools console, you’ll get the ability to place breakpoints or pause the execution. Pausing the execution may not work immediately, as an active JavaScript operation is required for that.

In Figure 1.6, you see how debugging a Node.js script in the Chrome DevTools can look:

Figure 1.6 – Debugging a Node.js script in the Chrome DevTools

In the preceding example, JavaScript is only run every minute. When the pause occurs, you should end up in the internal/timers.js part of Node.js itself. This is a different JavaScript file, but it’s part of the whole Node.js framework. The file can be integrated because it follows certain conventions and rules that are known as CommonJS.

One thing that Node.js got right from the beginning was to introduce an explicit way of obtaining and using functionality. JavaScript in the browser suffered from the global scope problem, which caused many headaches for developers.

Global scope

In JavaScript, the global scope refers to functionality that is accessible from every script running in the same application. On a website, the global scope is usually the same as the window variable. Attaching variables to the global scope may be convenient and sometimes even necessary, but it may also lead to conflicts. For instance, two independent functions could both try to write and read from the same variable. The resulting behavior can then be hard to debug and very tricky to resolve. The standard recommendation is to avoid using the global scope as much as possible.

The idea that other functionalities are explicitly imported was certainly not new when Node.js was introduced. While an import mechanism existed in other programming languages or frameworks for quite some time, similar options have also been available for JavaScript in the browser – via third-party libraries such as RequireJS.

Node.js introduced its module system with the name CommonJS. The basis for Node.js’s implementation was actually a project developed at Mozilla. In that project, Mozilla worked on a range of proposals that started with non-browser use but later on expanded to a generic set of JavaScript specifications for a module system.

CommonJS implementations

Besides the implementation in Node.js, many other runtimes or frameworks use CommonJS. As an example, the JavaScript that can be used in the MongoDB database leverages a module system using the CommonJS specifications. The implementation in Node.js is actually only partially fulfilling the full specification.

A module system is crucial for allowing the inclusion of more functionality in a very transparent and explicit manner. In addition to a set of more advanced functionality, a module system gives us the following:

• A way of including more functionality (in CommonJS, via the global require function)
• A way of exposing functionality, which can then be included somewhere else (in CommonJS, via the module-specific module or exports variables)

At its core, the way CommonJS works is quite simple. Imagine you have a file called a.js, which contains the following code:

const b = require('./b.js');
console.log('The value of b is:', b.myValue);

Now the job of Node.js would be to actually make this work, that is, give the b variable a value that represents the so-called exports of the module. Right now, the script would error out saying that a b.js file is missing.

The b.js file, which should be adjacent to a.js, reads as follows:

exports.myValue = 42;

When Node.js evaluates the file, it will remember the defined exports. In this case, Node.js will know that b.js is actually just an object with a myValue key with a value of 42.

From the perspective of a.js, the code can therefore be read like this:

const b = {
  myValue: 42,
};
console.log('The value of b is:', b.myValue);

The advantage of using the module system is that there is no need to write the outputs of the module again. The call to require does that for us.

Side effects

Replacing the call to require with the module’s outputs is only meant for illustrative purposes. In general, this cannot be done as the module evaluation can have some so-called side effects. A side effect happens when implicit or explicit global variables are manipulated. For instance, already writing something to the console or outputting a file in the module evaluation is a side effect. If we’d only replace the require call with the imported module’s exports, we would not run the side effects, which would miss a crucial aspect of the module.

In the given example, we used the name of the file directly, but importing a module can be more subtle than that. Let’s see a refined version of the code:

a.js

const b = require('./b');
console.log('The value of b is:', b.myValue);

The call to./b.js has been replaced by ./b. This will still work, as Node.js will try various combinations for the given import. Not only will it append certain known extensions (such as .js) but it will also look at whether b is actually a directory with an index.js file.

Therefore, with the preceding code, we could actually move b.js from a file adjacent to a.js to an index.js file in the adjacent directory, b.

The greatest advantage, however, is that this syntax also allows us to import functionality from third-party packages. As we will explore later in Chapter 2, Dividing Code into Modules and Packages, our code has to be divided into different modules and packages. A package contains a set of reusable modules.

Node.js already comes with a set of packages that don’t even need to be installed. Let’s see a simple example:

host.js

const os = require('os');
console.log('The current hostname is:', os.hostname());

The preceding example uses the integrated os package to obtain the current computer’s network name.

We can run this script with node in the command line:

$ node host.js

The current hostname is: DESKTOP-3JMIDHE

This script works on every computer that has Node.js installed.

In this chapter, we discovered Node.js for the first time. You should now have a good idea of the core principles (such as event loop, threads, modules, and packages) upon which Node.js was built. You have read a bit about Node.js’s history and why V8 was chosen as the JavaScript engine.

One of the key things to take away from this chapter is how the event loop works. Note that part of this knowledge is not exclusive to Node.js. The distinction between micro tasks and tasks is an integral part of how JavaScript engines, even the JavaScript engine of your browser, work.

Lastly, you are now equipped to use the node command-line application, for example, to run or debug simple scripts, which can export and import functionality using the CommonJS module system. You learned how to use the Chrome web browser to inspect Node.js scripts as you can with websites.

In the next chapter, we will increase our knowledge about CommonJS by learning how we can efficiently divide code into modules and packages.