Shaping Data with Grafana Transformations

Now that you understand how to connect data source queries to visualizations, we’re going to take a step back and look at one of the key features in Grafana’s visualization pipeline: the DataFrame. A DataFrame is an object that contains data received from a data source query and provides the source data for visualization.

In this chapter, we will learn more about DataFrames, their role in how Grafana visualizes data, and how to manipulate them using Grafana’s transformation operators. We will cover the following topics:

• About Grafana DataFrames and transformations
• Exploring the various transformation functions
• Expanding analysis with a transformation
• Chaining transformations into a visualization pipeline

First, we will answer the question of what a Grafana DataFrame is, its role, and how transformation operators affect it. Next, we will look at the most useful of the transformation operators...

About Grafana DataFrames and transformations

While it is easy to imagine that Grafana simply takes the results of one or more data source queries, and somehow feeds them to the panel visualization of your choice, the reality is a bit more complex than that.

Every query result from the Query tab is managed separately as an independent data series by Grafana. That way, you have full control over each series and how it may be displayed in a panel visualization.

Next, each of those datasets is packaged into a single object Grafana referred to as a DataFrame. If you are at all familiar with Excel or Google spreadsheets from the financial world, or pandas or Spark from the data science world, you already have experience with the concept.

A Grafana DataFrame is, like similar objects, a rows and columns (technically an array of arrays) data structure, with each row a combination of one or more columnar fields. Because the DataFrame is a consistent data structure, it can easily be...

Exploring the various transformation functions

Now that we’ve looked at some of the capabilities of the transformation functions, let’s take a closer look at some of the more commonly used transformations. Before we do that, however, let’s identify the UI for the Transform tab to familiarize ourselves with the controls. This is one of the typical transformations in action:

Figure 6.1 – Transformation UI

Besides the + Add transformation button, the Transform tab’s interface bears some similarities to the Query tab’s interface. Let’s explore the UI, which is shown in the preceding screenshot with its main areas numbered for reference:

1. The UI for setting the transformation parameters, which will vary from one to the next.
2. Various controls for managing the transformation (from left to right):
   • Shows/hides help information about the transformation
   • Opens a debug window showing the data objects before...

Expanding analysis with a transformation

Let’s look at a relatively simple example of how we might use the transformation functions we’ve looked at to aid in analysis. We’ll add a few queries, each representing a different metric, and then we’ll use the Reduce transformation to create some aggregations of the data from each query, all in a tabular format.

The data we’ll be using is derived from the ETL we created in Chapter 5, Extracting and Visualizing Data with InfluxDB and Grafana. Feel free to copy over the code from the Chapter05 directory or use the Chapter06 directory; both contain the same code.

The process is virtually identical to what we covered in Chapter 5, Extracting and Visualizing Data with InfluxDB and Grafana, so we won’t go into the details. Here is the process in schematic format, so refer to Chapter 5, for the details:

1. Run the Python weather.py script to download the NOAA weather data from a series of weather...

Chaining transformations into a visualization pipeline

Now that we’ve worked out how to add a single transformation to a set of query data frames, let’s take it one step further and chain together a series of transformations to create a visualization pipeline. This example might seem a bit trivial, but it illustrates how to work through the process of manipulating tables via transformations to produce a result that facilitates a specific visualization.

In this case, we are going to take some of our weather data and make transformations to the query result, a use case that resembles a situation where you might not have the data in the format you want.

From there, we will do some transformations that will produce a dataset suitable for the time series visualization. The results will be along the lines of what we accomplished in the previous chapter, but the idea here is to get the results by transformation rather than by modifying the query. Remember, there may be...

Summary

We covered a lot in this chapter! We learned about the three components of the visualization pipeline – query, transformation, and visualization – and how the DataFrame is the key data structure that’s processed by the pipeline. We explored the purpose of transformations and the many capabilities of transformation functions. We explored several common transformation functions with example use cases. We also tried out real-world transformations both singly and multiply in transformation chains. Finally, we examined some limitations of transformations.

In the next two chapters, we’ll continue down the visualization pipeline and look at some of the more commonly used visualizations: the stat, gauge, and bar chart visualizations in Chapter 7Chapter 7, Surveying Key Grafana Visualizations; followed by the geomap and heatmap, and revisiting the table visualization in Chapter 8Chapter 8, Surveying Additional Grafana Visualizations. This will open up the possibility of...