Chapter 4. Making Data Useful

When creating visualizations for the Web, chances are that the format your data comes in will not be the final format you use with D3. We will take a look at making our datasets useful with both D3 and regular JavaScript.

We start with a quick dip into functional programming to bring everyone up to speed. A lot of this will be self-evident if you use Haskell, Scala, or Lisp, or write JavaScript in a functional style. Functional programming is a hot topic in JavaScript development right now, for good reason -- it makes your code easier to read, encourages good practices, such as not mutating variables, and leverages one of JavaScript's best features as a language --  the use of first-class functions. We'll take a look at what that means in a short while.

We will continue with loading external data in a variety of different ways, take a closer look at scales, and finish with some temporal and geographic data.

Thinking about data functionally

I've mentioned previously that D3 is very functionally designed, meaning that it uses some of the idioms JavaScript has adopted from functional programming. Although we can still approach D3 development in a very classical, object-oriented fashion, our lives will be much easier if we start thinking about our code and data with a functional mindset.

The good news is that JavaScript almost counts as a functional language; there are enough features to get the benefits of a functional style, and it also provides enough freedom to do things imperatively or in an object-oriented way. The bad news is that, unlike real functional languages, the environment gives no guarantee about our code.

Note

Later on, in Chapter 9, Having Confidence in Your Visualizations, we'll look at TypeScript, which allows compilation of JavaScript using static types, and Tern.JS , which analyzes code in order to improve tooling. These efforts go a great deal toward improving confidence in...

Scales

We've already used scales many times -- we had a chant, if you remember; what was that again?

SURPRISE POP QUIZ:

INPUT!: [ ] DOMAIN! [ ] NOT DOMAIN!
OUTPUT!: [ ] RANGE! [ ] NOT RANGE!

If you got INPUT! = DOMAIN! and OUTPUT! = RANGE!, you are totally correct!

The reason we use scales is to avoid math. This makes our code shorter, easier to understand, and more robust, as mistakes in high school mathematics are some of the hardest bugs to track down.

To reiterate a point I've hopefully been hammering home since Chapter 1, Getting Started with D3, ES2017, and Node.js, a function's domains are those values that are defined (the input), and the range are those values it returns. The following figure is borrowed from Wikipedia:

Here, X is the domain, Y is the range, and the arrows are the functions. We need a bunch of code to implement this manually:

   let shape_color = shape => { 
     if (shape == 'triangle') { 
       return 'red'; 
     } else if (shape == 'line') { 
       return 'yellow...

Loading data

Quite often, you won't have the benefit of being able to create data using the Math random number generator functions, you'll need to load it from an external source. While sometimes it's easier to have your code generate your dataset, most of the time you'll be mapping real data to what you create with D3.

You can get data in a number of ways. I'll cover the main ones here:

• Make some sort of manual HTTP request: We already did this in earlier chapters. We supply a URL to a function that causes the browser to make a request. Both XMLHttpRequest and Fetch fall into this category. To import JSON, the server needs to have Cross-Origin Resource Sharing (CORS) enabled, meaning that it sends a header resembling Access-Control-Allow-Origin: *. This is due to the browser's security model, but it doesn't apply if we're loading data off the same domain, so we're able to do that without any extra work.
• Import it as a module: Some datasets are available as modules via npm. A good example of...

Flow control

JavaScript is a single-threaded asynchronous language, meaning that it doesn't really fork in the same way you would expect a multi-threaded language like C++ to, and thus a function blocking the main thread causes everything to grind to a halt. Luckily, the asynchronous part of that means functions generally won't do this, and functions can be written so that the next one can fire before the first one finishes.

On the one hand, this is one of the most interesting and powerful aspects of JavaScript as a language. On the other, it makes things somewhat more difficult to reason about, and adds a degree of complexity to organizing one's code. While you can do pretty much anything without getting too bogged down in the JavaScript event model, the one place where the asynchronous nature of JavaScript is particularly visible is when you make a request -- no matter how fast broadband Internet gets, so long as it conforms to the standard model of physics, there will always be some degree...

Geography

Geospatial data types are used for weather or population data; anything where you want to draw a map. Converting real-world coordinates into something representable on a 2D plane is a complex mathematical problem that has spanned centuries of human history (and still isn't really solved in any way if the huge number of projections that ship with d3-geo-projection is any indication).

D3 gives us three tools for geographic data:

• Paths produce the final pixels
• Projections turn sphere coordinates into Cartesian coordinates
• Streams speed things up

The main data format we'll use is TopoJSON, a more compact extension of GeoJSON, created by Mike Bostock. In a way, TopoJSON is to GeoJSON what DivX is to video. While GeoJSON uses the JSON format to encode geographical data with points, lines, and polygons, TopoJSON instead encodes basic features with arcs and re-uses them to build more and more complex features. As a result, files can be as much as 80 percent smaller than when we use GeoJSON...

Summary

You've made it through the chapter on data!

We really got to the core of what D3 is about - that is, data wrangling. We've looked at the various types of JavaScript flow control, resolved more promises than we care to remember and converted some shapefiles to TopoJSON -- at this point, your ES2015 skills should be approaching (if they aren't already over) 9,000.

Lastly, we made a cool map that showed a particular set of data, where we mashed up multiple datasets to create something cool. You'll find most of the more interesting data visualizations you build involve combining multiple datasets -- with your new-found skills with promises, you're well on your way to mashing up all the data.

In the next chapter, we'll look at making our stuff pop visually by adding animations. Now things are starting to get pretty exciting!