Chapter 5. Loading and Parsing Data

In the previous chapters, you learned how to create a simple scatter chart directive with D3.js and include it in an AngularJS application. Until now, we just generated random data to plot in charts.

In this chapter, you will learn how to feed the visualization directive with real data. Therefore, we need to load raw data from an external resource, we need to parse it to JavaScript objects, and we need to group the data for the visualization. First, we need to load log files from a remote server. Therefore, we will take a look at different techniques to load data into the AngularJS application: XHR with D3.js and $http with AngularJS. The goal is to understand the advantages and disadvantages of these techniques and to know which data loading module suits the best for the visualization application.

Once we have the raw data in the format of a string on the client, we need to parse and process it in order to feed it into the JavaScript application. This means...

Loading external data

In a modern web application, it's a common task to load and reload data from external resources, regardless of whether we read from the database or plain text files. Thus, nearly every JavaScript framework includes its own functions to load external data, that is, in most of the cases a wrapper of the native XMLHttpRequest (XHR) object. In our application setup, we have the following options to load external data:

• XMLHttpRequest: This is a native XHR object provided by most modern browsers

• d3.xhr(): This is a wrapper function for the XMLHttpRequest object in D3.js

• $http: This is an Angular wrapper module for the XMLHttpRequest object

These implementations use the unidirectional XMLHttpRequest to request data from a web server. Unidirectional means that we can solely request data from the client and then wait for the response of the server. Thus, we also don't know if there is new data available on the server. If we want a "real-time-like" behavior, we will need to continuously...

$http – the Angular wrapper for XHR

If we are developing a component for AngularJS, then we should use all the benefits and advantages that this framework provides, such as Promises, caching, mocking, and so on. For XHR, AngularJS provides an easy-to-use function that implements Promises. Let's take a look at an example:

var url = "files/access.log";
$http.get(url)
.then(function(response){
  console.log(response.data);
});

Looks pretty neat, doesn't it? This is exactly why we want to use an abstraction provided by AngularJS. Now, we can make an HTTP request with all the advantages from the AngularJS world. For completeness, let's also look at the POST request:

var url = "files/access.log";
var data = {'test': 'my-data'};
$http.post(url, data)
.then(function(response){
  console.log(response.data);
});

Also, the preceding code will load the log file and print it as the previous examples.

Creating an AngularJS data loading component

Now, we will use the AngularJS implementation for the data loading...

Parsing log files to JavaScript objects

Once we load the string of log entries, we need to parse these entries to an array of data points in order to display them. We want to parse the plain text log file to an array of JavaScript objects.

In this section, I will show you a convenient way to mix all techniques and to retrieve the best of both worlds. We will implement a preprocessing service component. This uses regular expressions to split the text into an array and D3.js to parse time strings to dates.

Parsing log files step by step

Let's look at the structure of the log files again:

66.249.64.121 - - [22/Nov/2014:01:56:00 +0100] "GET /robots.txt HTTP/1.1" 302
507 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
66.249.64.129 - - [22/Nov/2014:01:56:01 +0100] "GET / HTTP/1.1" 302 487 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

From this string, we ultimately want to generate an array of JavaScript objects in the shape of [{ip:...

Displaying logs

In the previous sections, we wrote data loading, parsing, and grouping services; now, it's time to use them in a simple example to display the logs. We want to display the number of visitors of our application in an interval of 5 minutes. We define the appearance of the logs directly in the controller as follows:

/* src/app.js */
// Application Module
angular.module('myApp', ['myChart'])
// Main application controller
.controller('MainCtrl', ["$scope", "d3", "SimpleHttpLoader", "StringParser", "Classifier", function ($scope, d3, SimpleHttpLoader, StringParser, Classifier) {

  // Formats date strings 22/Nov/2014:01:56:00 +0100
  var formatter = d3.time.format("%d/%b/%Y:%H:%M:%S %Z");
  $scope.log = {
    // Source of the log file
    src: 'files/access2.log',

    // Data entries
    data: [],
    // Maps response array to readable JSON object
    map: function(d) {
      return {
        time: +formatter.parse(d[2]),
        ip: d[0],
        request: d[3],
        status...

Summary

In this chapter, you learned about two different XHR techniques to load external data for the visualization: d3.xhr with D3.js and $http with AngularJS. If we include an already existing chart application in the AngularJS application, we can wrap the D3.js loading component into a service. In order to test this component, we need to write asynchronous unit tests.

If we write the visualization from scratch, we'd better implement the data loading technique with the $http module of Angular JS. This allows you to mock the behavior of $httpBackend and automatically serve data for specified routes. With this method, we can continue to write synchronous tests, which are more readable and expressive.

In the second part of this chapter, we parsed a string of log entries into an array of JavaScript objects. In addition to the parser service, we implemented a grouping service based on the d3.nest() function. In the end, we brought all the parts together and plotted an aggregated view of the log...