Visualizing Streaming Data

In the previous chapters, we've discussed most of the standard visualizations and components provided by D3. In this chapter, we're going to look at a specific implementation where we visualize streaming data, using a number of the building blocks we've seen in the previous chapters.

In this chapter, we're going to create the following visualizations:

• We're going to start very simply by just creating a line graph which is fed by mouse data. This data represents the streaming data that we want to visualize. We'll use transitions to animate the line, drawn through the randomly created values, from right to left.
• In the second example, we're going to connect to a WebSocket which provides us with ECG and respiratory data. Whenever the WebSocket pushes an element to our visualization, we'll update the drawn lines.
• In the previous chapters, we've...

Simple streaming line

For our first example, we're going to create two simple streaming lines. The result we'll be aiming for is the following (which you can see by opening up the DVD3/src/chapter-06/D06-01.html example):

In this image, we have got two horizontal lines which move from the right of the screen to the left. Whenever you move your mouse, the distance traveled is stored and rendered as two lines. Each point in the graph represents the total distance traveled along the x-axis or the y-axis. The bottom of this graph is a simple axis which shows the current time, and which moves together with the line graph.

To accomplish this graph, we need to take the following steps:

1. The first thing we need to do is define the various scales which we can use for rendering the scale at the bottom, and which we use to determine the x and y positions of each graph element.
2. When we've got the scales, we...

Heart rate and respiratory monitoring

For this example, we're going to show data pushed from a backend server. Specifically, we're going to visualize heart rate and respiratory data which is pushed over a WebSocket to our D3 frontend.

We're going to recreate the following visualization (the DVD3/src/chapter-06/D06-02.html example):

To get this example up and running, we're going to take the following steps:

1. Get sample respiratory data which we can push to the frontend.
2. Create a simple WebSocket server, which pushes the information.
3. Make the D3 visualization which responds to information pushed from the server.

We start with the sample data.

Getting the sample data

PhysioNet is an online research platform which provides free access to a large...

Creating the visualization

This visualization will have the following features:

• We're going to draw a line showing the ECG and respiratory information
• We load two external SVG images, a heart and a set of lungs, and animate those based on the data we receive

We once again start with setting up the data structures and the scales.

Scales, lines, and array initialization

The first thing we need to do is define some variables:

var ecgAvg =  16800; 
var respAvg = 16000; 
var n = 800; 
 
var data = d3.range(n).map(function(d) {return { 
    "ecg": ecgAvg, 
    "resp": respAvg 
}});

We use the ecgAvg and respAvg to prefill an array with our initial data. This will allow us to start our visualization with an empty line, while new data comes in. The n...

Random data-driven streamgraph

When you draw a streamgraph, the data of the different series are put one on top of another, just like for an area chart. Instead of an area chart, though, the bottom of the chart isn't fixed to a single position, but fluctuates, providing a visually appealing way of showing different series:

For our example, we're going to create a streaming variant of this streamgraph. We're going to use a server which pushes random data over a WebSocket to the browser and, based on that data, updates the streamgraph and moves it to the left. The final result we're aiming for is the DVD3/src/chapter-06/D06-03.html example.

When you open this example, and you have the random WebSocket server running (more on that in the next section), you'll see the following streamgraph moving from right to left:

We start again by looking at the server side where we've created a simple...

Visualizing Meetup.com RSVP data on a map

For our last streaming example, we're going to connect our web frontend to the public streaming API of Meetup. With Meetup, people can easily schedule meetups with people that share interests. An interesting aspect of Meetup is that it offers a publicly available WebSocket endpoint which provides RSVP information on public meetings. When we connect to that endpoint, we get information back which looks like this:

Each message is a single RSVP from someone either wanting to attend a meeting, or someone letting the organizers know they won't attend. When we look a bit closer at the JSON message, we see a lot of useful information:

{ 
  "venue": { 
    "venue_name": "The Moon Under Water (Wetherspoons)", 
    "lon": -0.74885, 
    "lat": 52.04258, 
    "venue_id": 995244 
  }, 
  "member": { 
    "...

Summary

In this chapter, we've shown you different ways of working with streaming data. We started by showing the concepts of creating streaming graphs by creating a simple line graph that shows real-time information based on mouse movements. After that, we connected to a simple WebSocket server which provided us with ECG and respiratory information, and streamed the received data directly to two line graphs. We also provided a streaming alternative to area charts in the form of a streamgraph. With a streamgraph, we can create a visually pleasing visualization of multiple streams of streaming data. Finally, we connected to a publicly available WebSocket server and used the information from that server to update a map in real time. In the next chapter, we're going to look into two final visualization approaches using Voronoi diagrams and heatmaps.

...