This chapter will introduce the D3.js (data-driven document) JavaScript library, describing its main features, explaining how it works, and showing how it drives data to transform documents. D3 contains an integrated set of tools that will help you bind data to graphical elements in a web page, in order to create scatterplots, bar charts, line charts, hierarchical node-link diagrams, networks, chord diagrams, sunburst charts, thematic geographic maps, or any interactive data visualization you can imagine. But D3.js is also a huge library, and is famous for having a steep learning curve. The goal of this book is to provide a learning path that will help you grasp its fundamental data-driven concepts and become familiar with its essential API.

In this chapter, you will learn how to set up your environment and test it by creating a very simple D3 application. It also includes a general overview of D3’s architecture, describing the relationships between its many modules and a brief description of each one.

This chapter will outline the following topics:

• D3 data-driven documents
• Using D3
• Modules microlibraries

D3, which stands for data-driven documents, is an open source JavaScript library used to create interactive web-based data visualizations. It provides a mechanism that connects arbitrary data to document elements, allowing their appearance and behavior to be driven by the data. Created by Mike Bostock, Jeff Heer, and Vadim Ogievetsky in 2001, it's currently used in hundreds of thousands of websites and is one of the most popular JavaScript data visualization libraries in the world.

If you have ever used interactive data applications from large news web portals such as The New York Times, The Washington Post, or The Guardian, there is a great probability that it was a D3 application. You may have also used one of the many charting libraries that are based on D3.

D3.js is also free and open source. You can use it in any project, commercial or not. Its source code is distributed in GitHub and is maintained by an active community of developers worldwide.

Yes, D3 is a JavaScript library, but no, D3 is not a charting library. There are no ready-to-use templates to create bar, pie, or line charts, for example. To create one of these charts, you have to draw all the lines, curves, and rectangles yourself using open standards such as SVG or HTML Canvas. D3, however, will do most of the hard work for you. It’s not trivial to use pure SVG to draw a bar chart; you need to scale data values so they fit in the chart, then calculate where to place each bar, and finally, set the coordinates of each rectangle before drawing it. Using D3, starting with an array of data, you can render all the bars with half a dozen chained commands in a single line of code.

D3 is a data visualization library. There are layout generators for pie charts that compute angles, which you can then use to draw arcs for the slices. There are functions that take a flat object array and turn it into a hierarchically linked object structure with coordinates for each node. You can use that data to draw circles at each coordinate point and draw lines between two nodes, rendering a tree. But you can also use the data differently, it's up to you. D3 doesn't restrict your creativity in any way. It doesn't tie you to a proprietary framework. Everything is based on open web standards.

D3 is also not only a data visualization library. Visualization is provided by HTML, CSS, or SVG. D3 focuses on the data. It's actually a collection of integrated JavaScript tools for manipulating the data structures necessary to create data visualizations. The core of the library is a fluent API used to select and manipulate the DOM. It replaces the DOM and libraries such as JQuery. It includes the data-driven mechanism that gives D3 its name, allowing you to bind arbitrary data to DOM elements, and then perform style and attribute transformations based on that data. This API is also used to bind and dispatch events, and to generate animated transitions.

D3 also includes tools to load and parse different data formats, such as JSON and CSV, perform general data manipulation on objects and arrays; generate data sequences and random numbers, perform interpolation and locale formatting. The actual data visualization parts contain layout generators, scales, axis generators, map projections, shape generators, color schemes, and other tools that are applied to previously selected DOM nodes and data.

A simplified view of D3's architecture is illustrated as follows. As implied by the name of the library, it's the data that drives the documents that display D3 visualizations. By adding, changing, and removing data, you directly affect the way your chart appears on the screen:

Data is usually provided as a JavaScript array, either generated locally or loaded from an external file. A typical D3.js script uses CSS selectors to select HTML or SVG elements and binds them to individual data items, removing, updating, or appending graphical elements automatically, when necessary.

You use CSS selectors to select one or more elements to create what D3 calls a selection object, which can then be used to apply styling and change the attributes of the selected elements. Binding an array to a selection object will map each data item to an element in the selection. These data items can be accessed in callback functions used in methods that set values for styles, attributes, and transforms.

You can also declare a selection of elements that don't exist in the DOM. Binding this selection to a data array will automatically create one element for each data item. You can then use the data to provide content, styles, and attributes for the new elements.

D3 also keeps data and their elements in sync. When updating with a smaller dataset, elements in excess are placed in a separate array, so you can remove them. If the dataset grows, existing elements are updated and missing elements are created and appended to fit the available data.

All that you need to start using D3 can be found at d3js.org where you can download and install the library as a single JavaScript file, a collection of standalone microlibraries, a CDN link, or an NPM installation script. The site also includes the official documentation, which covers the library in great detail, with hundreds of tutorials for beginners and advanced users, and thousands of online examples.

The D3 site is a great place to start because you can try out several examples of data visualizations created with D3. Just click on any hexagon and it will lead you to a page showing a D3 application and its source code. Most are published in GitHub and published using the Bl.ocks platform (bl.ocks.org), which is a huge portfolio based on GitHub source code. Newer projects use the Observable platform (observablehq.com), which contains interactive tutorials where you can edit the code and see changes in real time. These platforms are very popular among the D3 community. Many of these tutorials were designed by creators and major contributors of the D3 library, such as Mike Bostock, Jason Davies, and Philippe Riviere.

Take some time to explore these examples and see what you can create using D3.

You don't need a sophisticated development environment to use D3. If you already develop Web applications using npm, you can install it with:

npm install d3

You can also add the d3.js file to a local folder after downloading the full library (default bundle) as a ZIP from the official website. Then you can import it to your HTML file using the <script> tag and a local relative path; for example (for the minified version):

<script src="js/d3/d3.v5.min.js"></script>

If you have a permanent web connection, it's probably simpler to use a CDN link:

<script src="https://d3js.org/d3.v5.min.js"></script>

For very simple applications that don't load external files, you can simply open the page in your browser from the file system. It's better to load the page using a web server. If you are using a code editor, it may already have a launch configuration or plugin that always starts a web server when you choose to load a page. You can also install a simple web server using npm, by using the following command:

npm install http-server –g

The preceding command will install it globally. Then, you can move to the directory where your files are located and simply type the following:

http-server

Then you can load your files from http://localhost:8080.

You can also develop D3 using online code editors, such as CodePen (codepen.io) or JSFiddle (jsfiddle.net). It’s also a great way to share your code.

Let's create a simple D3-powered data-driven visualization to test your configuration. Create a new HTML file in your development environment and import the D3 library, or import it into any HTML page using the <script> tag as shown in the last section. Then add an <svg> and a <script> block inside your page's <body> as follows:

<body>
   <svg id="chart" width="600" height="200"></svg>
    <script> 
    </script>
</body>

Now add the following array inside the <script> block:

const array = [100, 200, 300, 350, 375, 400, 500];

We will use that array to generate a series of dots. Type in the following code (you can ignore the comments), which consists of a series of chained commands:

d3.select("#chart")     // selects the svg element by id (like JQuery)
  .selectAll("circle")  // declares the elements we would like to create
  .data([100])          // sets the data to drive the creation of the 
                        // elements
  .enter()              // creates a selection to add elements per 
                        // data item
  .append("circle")     // appends an element of this type to each 
                        // data item
    .attr("r", 10)      // sets “r” attribute
    .attr("cy", 100)    // sets “cy” attribute
    .attr("cx", d => d) // sets “cx” attribute (same as function(d) { 
                        // return d; }

If your configuration is working correctly, when you load the page containing this code in a browser, you should see a single dot on the screen:

If you inspect the dot with your browser's JavaScript development tools, you will notice that the following code was generated inside the SVG:

<svg width="600" height="200">
    <circle r="10" cy="100" cx="100"></circle>
</svg>

The preceding JavaScript code selects the <svg> element from the page and then selects all <circle> elements inside it. But there aren’t any, so the selection is empty. The next line, however, contains a data() command with a one-element array: [100]. The enter() command binds the data to the selection creating a selection of <circle> placeholders the same size as the data array. In this case, it will only contain one placeholder. Finally, the append() command, called in the context of this selection, appends a <circle> element to it, making it a child of the <svg> element.

The last three commands set attribute values and the last one takes a function and returns the element received by the function, which is used as the attribute's value. This element is the value 100 that was stored in the array passed to the data() command.

The code is available in the HelloWorld/1-intro-d3.html file from the GitHub repository for this chapter.

Now let's make some changes. Replace the [100] in the data() command with array. It will now reference the seven-element array we created before:

.data(array)

Run the page again. What happened? Consider the following screenshot:

Now there are seven dots on the screen; the same number of dots as the data array. And each dot is positioned at a different horizontal position (which was defined by the cx property of the <circle>) that is set using values from the data array. We used the data array to drive the position of each circle.

Now skip a line and add this code after the selection (see HelloWorld/3-update.html):

setTimeout(function() {
    d3.select("#chart").selectAll("circle")
      .data([50, 75, 125, 225, 325, 425, 450])
        .attr("r", 5)
        .attr("cx", d => d)
        .style("fill", "red")
}, 2000)

This code will run two seconds after the page loads and shows the circles in the positions defined by the first chain of commands. The function inside it selects the #chart SVG element again, and then all the circles inside it. But this time, these circles do exist. The data() command binds a new data array to the selection. No enter() command is called because there aren't any elements to be added.

The attr() commands are used to the circle's attributes. Only two attr()commands were called in the preceding code because we are only changing two attributes: the radius of the circle, and the position, which will now obtain its value from the new array. Besides that, we also used style() to change the color of the dots. If you run this file, two seconds later, the dots shrink, move to the left and become red:

One very nice feature of D3 is how simple it is to animate transitions. You just need to chain a transition() command before the attributes and styles that changed. The default transition takes a quarter of a second. Let's make it a little longer by configuring duration. Add the following line between the data() command and the first attr(), to add a one-second transition during the data update:

.transition().duration(1000)

Your page body should now look like this (see HelloWorld/4-transition.html):

<body>
<svg id="chart" width="600" height="200"></svg>
<script>
    const array = [100, 200, 300, 350, 375, 400, 500];

    d3.select("#chart")
            .selectAll("circle")
            .data(array)
            .enter()
            .append("circle")
            .attr("r", "10")
            .attr("cy", 100)
            .attr("cx", d => d)

    setTimeout(function() {
        d3.select("#chart").selectAll("circle")
          .data([50, 75, 125, 225, 325, 425, 450])
          .transition().duration(1000)
            .attr("r", 5)
            .attr("cx", d => d)
            .style("fill", "red")
    }, 2000)
</script>
</body>

Now, after two seconds, the dots will spend another second changing color, shrinking, and moving to the left. Congratulations! You created a full D3 application, complete with data updates and transitions.

Although you don't need a full frontend modular development environment to create visualizations with D3.js, you still need a good debugger. Every browser comes with development tools that allow you to navigate a static page structure and generated DOM elements, and a console where you can interact in real time with the data used by the JavaScript engine in the real time.

The most important tool is the JavaScript console, where you will see any error messages. It's very common to get a blank page when you expected something else and to not have a clue on why your code doesn't work as expected. Sometimes it's just a comma you forgot, or the internet is down and some file was not loaded. If you have the JavaScript console open while you run your page, it will instantly tell you what's going on. It's also a good idea to use an editor with line numbering since most error messages inform the lines where the problem occurred:

You can open the developer tools as a frame in your browser or as a separate window. Following are the menu paths for the JavaScript console in latest versions of the three most popular browsers:

• Chrome: View | Developer | JavaScript Console
• Firefox: Tools | Web Developer | Web Console
• Safari: Develop | Show Error Console

Many code fragments and examples can be tested by simply typing them in the JavaScript console. The console’s context is the currently loaded page. You will be able to access the functions of any JavaScript library file that was loaded with the <script> tag, and any global variables declared in <script></script> blocks, so you can also use the console to experiment with D3. The console is a great way to learn D3 since you can run functions in real time and immediately see the results they produce. You can try out many code examples in this book using the JavaScript console.

You don't have to always load the entire D3 library. D3 is a modular library, so if you are only using some D3 features, you can include only the parts that you are using. The minified default bundle is about 240 KB in size, but you may be able to create your chart using as little as 13 KB if you don't need many modules. An animated interactive bar chart, complete with tooltips, transitions, colors, and SVG graphics can be created with less than 24 KB loading just the following two modules:

<script src="https://d3js.org/d3-selection.v1.min.js"></script>
<script src="https://d3js.org/d3-transition.v1.min.js"></script>

But if you need axes, maps, and other features, you will require more modules and dependencies. In this case, either use the default bundle, or set up a development environment where you can install each module using npm, since it automatically includes any dependencies. For production, you can generate and export a custom bundle using a packing tool such as Rollup. To install any module using npm, use the following:

npm install module-name

For the examples in this chapter (and most of the book) we will import D3 using the <script> tag with the CDN URL to the default bundle.

Even if you always use the default d3.js bundle, you should be aware of the modules it contains, and of the modules that are not part of the default bundle, because they need to be imported separately in case you require their functions. All the official documentation is also organized per module, and knowledge of the modular structure will allow you to tune the performance of your D3 application, in case you decide to create a custom bundle. Modules have their own versioning systems. You might need to load a module separately if you wish to use a feature included in a new major version, not yet included in the default bundle.

The following diagram shows the modules available in D3.js version 5, indicating direct dependencies (transitive dependencies are not shown). For example, if you need to use a function from the d3-scale module, you should import all the direct dependencies: (d3-array, d3-time-format, d3-collection, d3-format, d3-interpolate) and the transitive ones:( d3-time and d3-color). In this case, you should either use npm or import the default bundle.

The following tables contain a quick reference of all modules available in the current version, classified according to their purpose. In this book, we will be using functions from almost all of them.

The following modules listed are used to generate, manipulate, transform, parse, and format data, in the form of numbers, text, arrays, objects, and files. They are all included in the default d3.js bundle:

These are core modules in D3 used to select and manipulate HTML or SVG elements by providing a concise API to the DOM. With these modules, you can select and filter elements (using CSS selectors), create elements, append, insert, or remove from the DOM tree, add attributes and contents, change styles or classes, connect event handlers, and join data. Practically any D3 application uses at least d3-selection.

The modules listed in the following table are used in dynamic visualizations when updating data, zooming, dragging, selecting and clicking charts and maps.

The following table lists modules that contain representations for machine-friendly and human-friendly color spaces and color schemes.

These modules are used to load files using Ajax.

The following modules listed contain a complete two-dimensional graphical API that can be used to create any type of visualization, from simple line, area, or pie charts to arbitrary paths and shapes.

The following table lists modules that are used to display information in geographic maps. They contain methods for spherical trigonometry, geographic projections, and other utilities.

The following modules listed, include generator functions, utilities, and algorithms to create visualizations for complex relationships such as node-link hierarchies, graphs, trees, networks, flow diagrams, tiles, and Voronoi diagrams.

This chapter provided a general introduction to the D3.js library, describing its architecture, modules, and showing how to set up a small D3 application that, although very simple, demonstrated one of the central paradigms in D3, which is driving the appearance of a visualization using arbitrary data.

The next chapter will consist of reference topics that cover fundamental technologies used by D3, such as SVG (an introductory tutorial), JavaScript (mostly a review of the fundamental data structures in ES 2015), and Canvas (a short reference). There is also a short section on data formats. If you are comfortable with all these topics, or if you want to start using D3 right away, you can skip them now and proceed to straight Chapter 3, Quick Start.

• D3.js documentation: d3js.org
• D3.js API reference: github.com/d3/d3/blob/master/API.md
• D3.js module repository: github.com/d3
• D3.js gallery: github.com/d3/d3/wiki/Gallery
• D3.js wiki: github.com/d3/d3/wiki
• Bl.ocks portfolios: bl.ocks.org
• Observable notebooks: observablehq.com