Data Visualization with D3 4.x Cookbook - Second Edition

4 (1 reviews total)
By Nick Zhu
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Getting Started with D3.js

About this book

This book gives you all the guidance you need to start creating modern data visualizations with D3 4.x that take advantage of the latest capabilities of JavaScript.

The book starts with the basic D3 structure and building blocks and quickly moves on to writing idiomatic D3-style JavaScript code. You will learn how to work with selection to target certain visual elements on the page, then you will see techniques to represent data both in programming constructs and its visual metaphor. You will learn how map values in your data domain to the visual domain using scales, and use the various shape functions supported by D3 to create SVG shapes in visualizations.

Moving on, you’ll see how to use and customize various D3 axes and master transition to add bells and whistles to otherwise dry visualizations. You’ll also learn to work with charts, hierarchy, graphs, and build interactive visualizations. Next you’ll work with Force, which is one of the most awe-inspiring techniques you can add to your visualizations, and you’ll implement a fully functional Choropleth map (a special purpose colored map) in D3.

Finally, you’ll learn to unit test data visualization code and test-driven development in a visualization project so you know how to produce high-quality D3 code.

Publication date:
February 2017
Publisher
Packt
Pages
370
ISBN
9781786468253

 

Chapter 1. Getting Started with D3.js

In this chapter, we will cover:

  • Setting up a simple D3 development environment

  • Setting up an NPM-based D3 development environment

  • Understanding D3-style functional JavaScript

 

Introduction


This chapter is designed to get you up and running with D3.js and covers fundamental aspects, such as what D3.js is and how to set up a typical D3.js data visualization environment. One particular section is also devoted to covering some lesser known areas of JavaScript that D3.js relies heavily on.

What is D3? D3 refers to Data-Driven Documents, and according to the official D3 Wiki:

D3 (Data-Driven Documents or D3.js) is a JavaScript library for visualizing data using web standards. D3 helps you bring data to life using SVG, Canvas and HTML. D3 combines powerful visualization and interaction techniques with a data-driven approach to DOM manipulation, giving you the full capabilities of modern browsers and the freedom to design the right visual interface for your data.

-D3 Github Wiki (2016, August)

In a sense, D3 is a specialized JavaScript library that allows you to create amazing data visualizations using a simple (data driven) approach by leveraging the existing Web standards. D3.js was created by Mike Bostock (https://bost.ocks.org/mike/ ) and superseded his previous work on a different JavaScript data visualization library called Protovis. For more information on how D3 was created and on the theory that influenced both Protovis and D3.js, please check out the links in the following information box. Here, in this book, we will focus more on how to use D3.js to power your visualization. Initially, some aspects of D3 maybe a bit confusing due to its different approach for data visualization. I hope that over the course of this book, a large number of topics, both basic and advanced, will make you comfortable and effective with D3. Once it is properly understood, D3 can improve your productivity and expressiveness with data visualizations by orders of magnitude.

Note

For a more formal introduction to the idea behind D3, refer to the Declarative Language Design for Interactive Visualization paper published by Mike Bostock on IEEE InfoVis 2010 at http://vis.stanford.edu/papers/protovis-design . If you are interested to know how D3 came about, I recommend that you check out the D3: Data-Driven Document paper published by Mike Bostock, Vadim Ogievestsky, and Jeffery Heer on IEEE InfoVis 2011 at http://vis.stanford.edu/papers/d3 . Protovis, the predecessor of D3.js, also created by Mike Bostock and Jeff Heer of the Stanford Visualization Group can be found at https://mbostock.github.io/protovis/ .

 

Setting up a simple D3 development environment


The first thing you will need when you start a D3-powered data visualization project is a working development environment. In this recipe, we will show you how a simple D3 development environment can be set up within minutes.

Getting ready

Before we start, make sure that you have your favorite text editor installed and ready on your computer.

How to do it...

We''ll start by downloading D3.js through the following steps:

  1. Download the latest stable version of D3.js from https://d3js.org/ . You can download the archived, older releases from https://github.com/d3/d3/tags . Additionally, if you are interested in trying out the bleeding edge D3 build on master branch, then you can fork https://github.com/d3/d3 .

  2. Once it is downloaded and unzipped, you will find two D3 JavaScript files, d3.js and d3.min.js, and other informational files in the extracted folder. For development purpose, it is recommended that you use the d3.js file, the non-uglified (minimized) version, since it can help you trace and debug JavaScript inside the D3 library. Once extracted, place the d3.js file in the same folder with an index.html file containing the following HTML:

        <!-- index.html --> 
         <!DOCTYPE html> 
         <html> 
         <head> 
             <meta charset=""utf-8""> 
             <title>Simple D3 Dev Env</title> 
             <script type=""text/javascript"" src=""d3.js""></script> 
         </head> 
         <body> 
     
         </body> 
         </html> 
    

This is all you need to create, in its simplest form, a D3-powered data visualization development environment. With this setup, you can essentially open the HTML file using your favorite text editor to start your development and also to view your visualization by opening the file in your browser.

How it works...

D3 JavaScript library is very self-sufficient. It has no dependency on any other JavaScript library except what your browser already provides.

Note

If your visualization's target browser environment includes Internet Explorer 9, it is recommended that you use the compatibility library Aight, which can be found at https://github.com/shawnbot/aight , and Sizzle selector engine, which can be found at http://sizzlejs.com/ .

Having the following character encoding instruction in the header section was critical before D3 v4 release since the older version of D3 used UTF-8 symbols, such as π, in its source; however, with D3 v4.x, it is no longer necessary. It is still considered a good practice however, since other JavaScript libraries you will include might be using UTF-8 symbols, as shown in the following example:

    <meta charset=""utf-8""> 

Note

D3 is completely open source under a custom license agreement created by its author Michael Bostock. This license is pretty similar to the popular MIT license, with only one exception where it explicitly states that Michael Bostock's name cannot be used to endorse or promote products derived from this software without his permission.

There's more...

Throughout this cookbook, numerous recipe code examples will be provided. All example source code is provided and hosted on GitHub (https://github.com/ ), a popular open source social coding repository platform.

How to get source code

The easiest way to get all the recipe source code that you will need is to clone the Git repository (https://github.com/NickQiZhu/d3-cookbook-v2 ) for this book. If you are not planning to set up a development environment for the recipes, then you can safely skip this section.

Note

In case you are not familiar with Git, its clone concept is similar to the checkout concept in other version control software. However, cloning does a lot more than simply checking out the files. It also copies all branches and histories to your local machine, effectively cloning the entire repository to your local machine so you can work even when you are completely offline with this cloned repository in your own environment.

First, install a Git client on your computer. You can find a list of Git client software at https://git-scm.com/downloads , and a detailed guide on how to install it on different operating systems at https://git-scm.com/book/en/Getting-Started-Installing-Git .

Note

Another popular way to get Git and GitHub working is to install the GitHub client, which gives you a richer set of features than simply Git. However, at the time of writing this book, GitHub only offered client software for Windows and Mac OS; refer to https://desktop.github.com/ .

Once the Git client is installed, simply issuing the following command will download all recipe source code to your computer:

> git clone [email protected]:NickQiZhu/d3-cookbook-v2.git
 

Setting up an NPM-based development environment


The simple setup demonstrated in the previous recipe is enough for implementing most recipes in this book. However, when you work on a more complex data visualization project that requires the use of a number of JavaScript libraries, the simple solution we discussed before might become a bit clumsy and unwieldy. In this section, we will demonstrate an improved setup using Node Packaged Modules (NPM), a de facto JavaScript library repository management system. If you are as impatient as me and want to get to the meaty part of the book, the recipes, you can safely skip this section and come back when you need to set up a more production-ready environment for your project.

Getting ready

Before we start, please make sure that you have NPM properly installed. NPM comes as part of the Node.js installation. You can download Node.js from https://nodejs.org/ . Select the correct Node.js binary build for your OS. Once installed, the following npm command will become available in your terminal console:

> npm -v 
2.15.8

The preceding command prints out the version number of your NPM client to indicate that the installation is successful.

How to do it...

With the NPM installed, we can now create a package descriptor file to automate some of the manual setup steps:

  1. First, under your project folder, create a file named package.json that contains the following code:

         { 
           ""name": "d3-project-template", 
           ""version": "0.1.0", 
           "description": "Ready to go d3 data visualization project template", 
           "keywords": [ 
             "data visualization", 
             "d3" 
           ], 
           "homepage": "<project home page>", 
           "author": { 
             "name": "<your name>", 
             "url": "<your url>" 
           }, 
           "repository": { 
             "type": "git", 
             "url": "<source repo url>" 
           }, 
           "dependencies": { 
               "d3":"4.x" 
           }, 
           "devDependencies": { 
               "uglify-js": "2.x" 
           } 
         } 
    
  2. Once the package.json file is defined, you can simply run the following:

             > npm install
    

How it works...

Most of the fields in the package.json file are for informational purpose only, such as its name, description, home page, author, and the repository. The name and the version fields will be used if you decide to publish your library into an NPM repository in the future. What we really care about, at this point, are the dependencies and devDependencies fields:

  • The dependencies field describes the runtime library dependencies that your project has, that is, the libraries your project will need to run properly in a browser.

  • In this simple example, we only have one dependency on D3. d3 is the name of the D3 library that is published in the NPM repository. The version number 4.x signifies that this project is compatible with any of the version 4 releases, and NPM should retrieve the latest stable version 4 build to satisfy this dependency.

Note

D3 is a self-sufficient library with zero external runtime dependency. However, this does not mean that it cannot work with other popular JavaScript libraries. I regularly use D3 with other libraries to make my job easier, for example, JQuery, Zepto.js, Underscore.js, and ReactJs to name a few.

  • The devDependencies field describes development time (compile time) of library dependencies. What this means is that libraries specified under this category are only required in order to build this project, and not required to run your JavaScript project.

Note

Detailed NPM package JSON file documentation can be found at https://docs.npmjs.com/files/package.json .

Executing the npm install command will automatically trigger NPM to download all dependencies that your project requires, including your dependencies' dependencies recursively. All dependency libraries will be downloaded into the node_modules folder under your project's root folder. When this is done, you can just simply create your HTML file as shown in the previous recipe, and load your D3 JavaScript library directly from node_modules/d3/build/d3.js.

The source code for this recipe with an automated build script can be found at https://github.com/NickQiZhu/d3-cookbook-v2/tree/master/src/chapter1/npm-dev-env .

Relying on NPM is a simple and yet more effective way to save yourself from all the trouble of downloading JavaScript libraries manually and the constant need for keeping them up to date. However, an astute reader may have already noticed that with this power we can easily push our environment setup to the next level. What if you are building a large visualization project where thousands of lines of JavaScript code will be created? Then obviously, our simple setup described here will no longer be sufficient. However, modular JavaScript development by itself can fill an entire book; therefore, we will not try to cover this topic since our focus is on data visualization and D3. In later chapters, when unit test-related recipes is discussed, we will expand the coverage on this topic to show how our setup can be enhanced to run automated build and unit tests.

Tip

D3 v4.x is very modular; so if you only need a part of the D3 library for your project, you can also selectively include D3 submodule as your dependency. For example, if you only need d3-selection module in your project, then you can use the following dependency declaration in your package.json file: "dependencies": {       "d3-selection":"1.x" }

There's more...

Although in previous sections it was mentioned that you can just open the HTML page that you have created using your browser to view your visualization result directly, this approach does have its limitations. This simple approach stops working once we  need to load data from a separate data file (this is what we will do in later chapters, and it is also the most likely case in your daily working environment) due to the browser's built-in security policy. To get around this security constraint, it is highly recommended that you set up a local HTTP server so your HTML page and the data file can be accessed from this server instead of being loaded from a local file system directly.

Setting up a local HTTP server

There are probably more than a dozen different ways to set up an HTTP server on your computer based on the operating system you use and the software package you decide to use to act as an HTTP server. Here, I will attempt to cover some of the most popular setups.

Python Simple HTTP server

This is my favorite for development and fast prototyping. If you have Python installed on your OS, which is usually the case with any Unix/Linux/Mac OS distribution, then you can simply type the following command in your terminal with Python 2:

> python -m SimpleHTTPServer 8888

Alternatively, type the following command with Python 3 distribution:

> python -m http.server 8888

This little python program will launch an HTTP server and start serving any file right from the folder where this program is launched. This is by far the easiest way to get an HTTP server running on any OS.

Note

If you don't have python installed on your computer yet, you can get it from https://www.python.org/getit/ . It works on all modern OS, including Windows, Linux, and Mac.

Node.js HTTP server

If you have Node.js installed, perhaps as part of the development environment setup exercise we did in the previous section, then you can simply install the http-server module. Similar to the Python Simple HTTP Server, this module will allow you to launch a lightweight HTTP server from any folder and start serving pages right away.

First, you need to install the http-server module using the following command:

> npm install http-server -g

The -g option in this command will install http-server module globally, so it will become available in your command-line terminal automatically. Once this is done, you can launch the server from any folder you are in by simply issuing the following command:

> http-server -p 8888

This command will launch a Node.js-powered HTTP server on the default port 8080, or if you want, you can use the -p option to provide a custom port number for it.

Note

If you are running the npm install command on Linux, Unix, or Mac OS, you may need to run the command in the sudo mode or as root in order to use the -g global installation option.

 

Understanding D3-style JavaScript


D3 is designed and built using functional style JavaScript, which might come as a bit unfamiliar or even alien to someone who is more comfortable with the procedural or object-oriented JavaScript styles. This recipe is designed to cover some of the most fundamental concepts in functional JavaScript required to make sense of D3, and furthermore enable you to write your visualization code in the D3 style.

Getting ready

Open your local copy of the following file in your web browser: https://github.com/NickQiZhu/d3-cookbook-v2/blob/master/src/chapter1/functional-js.html .

How to do it...

Let's dig a little deeper into the good part of JavaScript, the more functional side. Take a look at the following code snippet:

     function SimpleWidget(spec) { 
       var instance = {}; // <-- A 
      
       var headline, description; // <-- B 
 
       instance.render = function () { 
         var div = d3.select('body').append("div"); 
 
         div.append("h3").text(headline); // <-- C 
      
         div.attr("class", "box") 
         .attr("style", "color:" + spec.color) // <-- D 
           .append("p") 
           .text(description); // <-- E 
 
         return instance; // <-- F 
       }; 
 
       instance.headline = function (h) { 
         if (!arguments.length) return headline; // <-- G 
         headline = h; 
         return instance; // <-- H 
       }; 
                
       instance.description = function (d) { 
         if (!arguments.length) return description; 
         description = d; 
         return instance; 
       }; 
 
       return instance; // <-- I 
     } 
      
       var widget = SimpleWidget({color: "#6495ed"}) 
         .headline("Simple Widget") 
         .description("This is a simple widget demonstrating
           functional      javascript."); 
       widget.render(); 

The preceding code snippet generates the following simple widget on your web page:

How it works...

Despite its simplicity, the interface of this widget has this undeniable similarity to the D3 style of JavaScript. This is not by coincidence but rather by leveraging a JavaScript programming paradigm called functional objects. Like many interesting topics, this is another topic that can fill an entire book by itself; nevertheless, I will try to cover the most important and useful aspects of this particular paradigm in this section, so you can not only understand D3's syntax but you'll also be able to create your code in this fashion. As stated on D3's project Wiki, this functional programming style gives D3 much of its flexibility:

D3's functional style allows code reuse through a diverse collection of components and plugins.                                                                                                                                                                               -D3 Wiki (2016, August)

Functions are objects

Functions in JavaScript are objects. Like any other object in JavaScript, function object is just a collection of name and value pair. The only difference between a function object and a regular object is that function can be invoked and additionally associated with the following two hidden properties: function context and function code. This might come as a surprise and seem unnatural, especially if you come from a more procedural programming background. Nevertheless, this is the critical insight most of us need in order to make sense of some of the strange ways that D3 uses functions.

Note

JavaScript in its pre-ES6 form is generally not considered very object oriented; however, function object is probably one aspect where it outshines some of its other more object-oriented cousins.

Now, with this insight in mind, let's take a look at the following code snippet again:

  var instance = {}; // <-- A 
 
  var headline, description; // <-- B 
 
  instance.render = function () { 
    var div = d3.select('body').append("div"); 
 
    div.append("h3").text(headline); // <-- C 
 
    div.attr("class", "box") 
      .attr("style", "color:" + spec.color) // <-- D 
      .append("p") 
      .text(description); // <-- E 
 
    return instance; // <-- F 
  }; 

At the lines marked as A, B, and C, we can clearly see that instance, headline, and description are all internal private variables belonging to the SimpleWidget function object. While the render function is a function associated with the instance object which itself is defined as an object literal. Since functions are just objects, they can also be stored in an object/function, referred to by variables, contained in an array, and being passed as function arguments. The result of the execution of the SimpleWidget function is the returning of object instance at line I as follows:

function SimpleWidget(spec) { 
... 
  return instance; // <-- I 
} 

Note

The render function uses some of the D3 functions that we have not covered yet, but let's not pay too much attention to them for now since we will cover each of them in depth in the next couple of chapters. Also, they basically just render the visual representation of this widget and do not have much to do with our topic on hand.

Static variable scoping

Curious readers will probably be asking by now how the variable scoping is resolved in this example since the render function has seemingly strange access to not only the instance, headline, and description variables but also the spec variable that is passed into the base SimpleWidget function. This seemingly strange variable scoping is actually determined by a simple static scoping rule. This rule can be thought as the following: whenever the runtime searches for a variable reference, the search will be first performed locally. When a variable declaration is not found (as in the case of headline on line C), the search continues toward the parent object (in this case, the SimpleWidget function is its static parent and the headline variable declaration is found at line B). If still not found, then this process will continue recursively to the next static parent, so on and so forth, till it reaches a global variable definition; if it is still not found, then a reference error will be generated for this variable. This scoping behavior is very different from variable resolution rules in some of the most popular languages, such as Java and C#.

It might take some time to get used to; however, don't worry too much about it if you still find it confusing. With more practice and by keeping the static scoping, rule in mind, you will be comfortable with this kind of scoping in no time.

Note

One word of caution here, again for folks from Java and C# backgrounds, is that JavaScript does not implement block scoping. The static scoping rule we described only applies to function/object but not at the block level, as shown in the following code: for(var i = 0; i < 10; i++){   for(var i = 0; i < 2; i++){     console.log(i);   } } You might be inclined to think this code should produce 20 numbers. However, in JavaScript, this code creates an infinite loop. This is because JavaScript does not implement block scoping, so the variable i in the inner loop is the same variable i used by the outer loop. Therefore, it gets reset by the inner loop and can never end the outer loop.

This pattern is usually referred as functional when compared with the more popular prototype-based Pseudo-classical pattern. The advantage of the functional pattern is that it provides a much better mechanism for information hiding and encapsulation since the private variables, in our case, the headline and description variables, are only accessible by nested functions through the static scoping rule; therefore, the object returned by the SimpleWidget function is flexible yet more tamper-proof and durable.

If we create an object in the functional style, and if all of the methods of the object make no use of this, then the object is durable. A durable object is simply a collection of functions that act as capabilities.

-Crockfort D. 2008

Getter-setter function

Let's take a look at the following code; something strange has happened on line G:

instance.headline = function (h) { 
  if (!arguments.length) return headline; // <-- G 
  headline = h; 
  return instance; // <-- H 
}; 

You might be asking where this arguments variable on line G came from. It was never defined anywhere in this example. The arguments variable is a built-in hidden parameter that is available to functions when they are invoked. The arguments variable contains all arguments for a function invocation in an array.

Note

In fact, arguments is not really a JavaScript array object. It has length and can be accessed using an index; however, it does not have many of the methods associated with a typical JavaScript array object, such as slice or concat. When you need to use a standard JavaScript array method on arguments, you will need to use the following apply invocation pattern: var newArgs = Array.prototype.slice.apply(arguments);

This hidden parameter when combined with the ability to omit function argument in JavaScript allows you to write a function such as instance.headline with unspecified number of parameters. In this case, we can either have one argument h or none. Because arguments.length returns 0 when no parameter is passed, the headline function returns headline if no parameter is passed, otherwise it turns into a setter if parameter h is provided. To clarify this explanation, let's take a look at the following code snippet:

var widget = SimpleWidget({color: "#6495ed"}) 
    .headline("Simple Widget"); // set headline 
console.log(widget.headline()); // prints "Simple Widget" 

Here, you can see how the headline function can be used as both setter and getter with different parameters.

Function chaining

The next interesting aspect of this particular example is the capability of chaining functions to each other. This is also the predominant function invocation pattern that the D3 library deploys since most of the D3 functions are designed to be chainable to provide a more concise and contextual programming interface. This is actually quite simple once you understand the variable, parameter function concept. Since a variable-parameter function, such as the headline function, can serve as setter and getter at the same time, returning the instance object when it's acting as a setter allows you to immediately invoke another function on the invocation result, hence the chaining.

Let's take a look at the following code:

var widget = SimpleWidget({color: "#6495ed"}) 
  .headline("Simple Widget") 
  .description("This is ...") 
  .render(); 

In this example, the SimpleWidget function returns the instance object (as on line I).

function SimpleWidget(spec) { 
... 
    instance.headline = function (h) { 
        if (!arguments.length) return headline; // <-- G 
        headline = h; 
       return instance; // <-- H 
    }; 
... 
    return instance; // <-- I 
} 

Then, the headline function is invoked as a setter, which also returns the instance object (as on line H). The description function can then be invoked directly on its return, which again returns the instance object. Finally, the render function can be called.

Now, with the knowledge of functional JavaScript and a working ready-to-go D3 data visualization development environment, we are ready to dive into the rich concepts and techniques that D3 has to offer. However, before we take off, I would like to cover a few more important areas, how to find and share code and how to get help when you are stuck.

There's more...

Let's take a look at some additional helpful resources.

Finding and sharing code

One of the great things about D3 when compared with other visualization options is that it offers a wealth of examples and tutorials that you can draw your inspiration from. During the course of creating my own open source visualization charting library and the creation of this book, I had relied heavily on these resources. I will list some of the most popular options available in this aspect. This list is by no means a comprehensive directory, but rather a starting place for you to explore:

  • The D3 gallery (https://github.com/d3/d3/wiki/Gallery ) contains some of the most interesting examples that you can find online regarding D3 usage. It contains examples on different visualization charts, specific techniques, and some interesting visualization implementations in the wild, among others.

  • Christophe Viau's D3 Gallery (http://christopheviau.com/d3list/gallery.html ) is another D3 gallery with categorization that helps you find your desired visualization example online quickly.

  • The D3 tutorials page (https://github.com/d3/d3/wiki/Tutorials ) contains a collection of tutorials, talks, and slides created by various contributors over time that demonstrates in detail how to use different D3 concepts and techniques.

  • D3 plugins can be found at https://github.com/d3/d3-plugins . Maybe some features are missing in D3 for your visualization needs? Before you decide to implement your own, make sure that you check out the D3 plugin repository. It contains a wide variety of plugins that provide some of the common and, sometimes, uncommon features in the visualization world.

  • The D3 API (https://github.com/d3/d3/blob/master/API.md ) is very well documented. This is where you can find detailed explanations for every function and property that the D3 library has to offer.

  • Mike Bostok's Blocks (http://bl.ocks.org/mbostock ) is a D3 example site, where some of the more intriguing visualization examples can be found and is maintained by its author Mike Bostock.

  • JS Bin (http://jsbin.com/ugacud/1/edit ) is a prebuilt D3 test and experiment environment completely hosted online. You can easily prototype a simple script using this tool or share your creation with other members in the community.

  • JS Fiddle (http://jsfiddle.net/qAHC2/ ) is similar to JS Bin; it also is a hosted-online JavaScript code prototyping and sharing platform.

How to get help

Even with all the examples, tutorial, and cookbook such as this, you might still run into challenges when you create your own visualization. The good news here is that D3 has a broad and active support community. Simply googling your question can most often yield a satisfying answer. Even if it doesn't, don't worry; D3 has a robust community-based support as the following:

About the Author

  • Nick Zhu

    Nick Zhu is a professional programmer and data engineer with more than a decade experience in software development, big data, and machine learning. Currently, he is one of the founders and CTO of Yroo.com - meta search engine for online shopping. He is also the creator of dc.js—a popular multidimensional charting library built on D3.

    Browse publications by this author

Latest Reviews

(1 reviews total)
I wasn't able to find this title through the Packt website; the search feature wasn't able to locate it. I had to use Google to find it.

Recommended For You

Book Title
Access this book, plus 7,500 other titles for FREE
Access now