Chapter 3. Loading, Filtering, and Grouping Data

In the previous chapter, you learned to draw a simple bar chart. In this chapter, you will learn to load, parse, and display real data from a remote location with this chart. In this chapter, you will learn the following:

• Why we need to preprocess our data

• How to filter outliers

• How to map data to another representation

• How to aggregate information

• How to load data using D3

• How to parse strings using Regular Expressions

• How to parse date strings

• How to format numbers

• How to group flat data

First, we will start with some basic preprocessing steps that are always necessary. We will use the filter function to filter outliers and unexpected values from the dataset; we will also use the map function to access inner properties of array elements.

Then, you will learn about reduce functions and how they can help in aggregating and extracting information out of the data. We will implement our own simple reduce function and also take a look at the ones provided...

Preprocessing data

In this chapter, you will learn about preprocessing and that it is an important step when you deal with real-world data. If we want to design a robust visualization that can handle different remote data sources, we need to make sure we process the data beforehand. Datasets can have outliers, can be noisy, can have different representations, can have nested objects, can be flat, and so on. You can see that this list is very long.

Thus, whenever we deal with real data, we need to process it before using it in a visualization; this step is called preprocessing.

Filtering data to remove outliers

As a first preprocessing step, we will look at filters. Filters are array operators that return an array containing a subset of the original elements. In the following figure, we see an example of such an operation:

Filtering a dataset

Let's try an example. We load the following array from a remote data source:

var data = [
  {x: 1, y: 6},
  {x: 2, y: undefined},
  {x: 3, y: 4},
  {x: 4...

Loading and parsing remote data

In this section, we want to visualize real data rather as in sample dataset; hence, we need to understand how to load data from remote locations. In JavaScript, you can use the native XMLHttpRequest (XHR) object to fetch data within your application. D3 implements not only a wrapper for the native XHR call, but it also implements wrappers for loading different data types and parsing them automatically.

Here is a list of some high-level wrappers in D3:

• d3.text(url[, mimeType][, callback]): This loads a plain text file with a mimeType from an URL; the data is handed over as a string to the callback where the parsing must be done manually

• d3.json(url[, callback]): This loads and parses a JSON file from an URL; the data is passed as an object in the callback

• d3.xml(url[, mimeType][, callback]): Loads and parses an XML file from an URL, the data is passed as an object in the callback

• d3.html(url[, callback]): This loads and parses an HTML file from an URL; the...

Grouping data

Most of the time when we find data resources on the Internet, it has a flat data structure just as a comma separated list. But often, it contains hierarchical structured data. In D3, it is very easy to group data together by creating nested tree structures of the dataset. To achieve this, we can use the d3.nest() operator. This operator creates a nest object, which can further define a grouping key via the .key() method and a sorting function via the .sortKey() method. These functions can also be repeated to create multiple nested layers of the flat data structures. Let's take a look at the dataset:

var values = [
  {
    date: "04/23/12",
    key: "Group1",
    value: "37"
  },
  {
    date: "04/23/12",
    key: "Group2",
    value: "12"
  },
  {
    date: "04/23/12",
    key: "Group3",
    value: "46"
  }, ...
];

First, we create a nest function that will then group our data accordingly:

var nest = d3.nest()
  .key(function(d, i){
    return d.date;
  });

A nest now implements...

Summary

In this chapter, you learned about data preprocessing in order to make it displayable for the visualization. We saw how to filter unexpected data, how to map data to a different representation, and how to aggregate data.

We took a look at loading data from remote resources and custom-parsing via Regular Expressions. We saw how to split a string into an array of elements that can then be used for further processing. In the end, you learned how to use D3 for grouping flat data into nested objects. This helped us to ultimately extract all the desired information out of the remote dataset.

In the next chapter, we will take the previously generated chart and make it responsive. You will learn what it exactly it means for a chart to be responsive and how we can use Bootstrap to achieve this.