Working with Advanced Dashboard Features and Elasticsearch

By now, you’re probably feeling comfortable with Grafana but have legitimate concerns about the effort involved. You may be thinking that the possibility of writing a lot of code to handle extract, transform, and load (ETL) tasks might eat into your time budget to build the dashboards. Perhaps the number of panels you will have to configure and organize on multiple dashboards seems potentially tedious, error-prone work.

In this chapter, we’re going to look at how to reduce the ETL burden using off-the-shelf tools, as well as how to use templates to fill a dashboard with variants, using only a single panel. We’ll also show you how annotations make it possible to drill down into aggregated data in order to examine individual data points. Then, we’ll take our dashboards and link them together with simple UI elements. Finally, we’ll look at strategies to share our dashboards with others.

...

Technical requirements

The tutorial code, dashboards, and other helpful files for this chapter can be found in this book’s GitHub repository at https://github.com/PacktPublishing/Learn-Grafana-10/tree/main/Chapter10.

Building the data server

Imagine for a moment that you are working for the public works department of a major city. Throughout the day, citizens use their phones and computers to report problems via the 311 service (https://www.open311.org). You’ve been tasked with accessing the 311 data, building dashboards, and presenting them to various stakeholders within the city government. They will want to see how many of the various types of calls are made to the system, as well as how they are distributed across the city in various council districts.

Before we can build our dashboards, we’ll need to get some data. Luckily, many major cities make anonymized 311 data publicly accessible in many popular data formats, including JSON and CSV. For this exercise, we’ll be working with 311 data from the city of San Francisco. This data is available via their extensive data portal at https://data.sfgov.org/City-Infrastructure/Current-FY-Cases/iy63-pi3t.

To get started with...

Connecting Grafana to Elasticsearch

Now that we have our data loaded into Elasticsearch, we’ll need to create a data source connection in Grafana to read it. Connecting to Elasticsearch is not substantially different from the InfluxDB data sources we’ve been using. The authorization controls for our Elasticsearch container allow open access to the data source, so we won’t need to use an API token to access it:

1. Open your browser to the Grafana app and select Connections | Add new connection from the main menu.
2. Search for the Elasticsearch data source and select it.
3. Click on Add a new data source. Fill out the following form fields:
   • Name: Elasticsearch
   • HTTP | URL: http://elasticsearch:9200
   • Elasticsearch details | Index name: data-index
   • Elasticsearch details | pattern: No pattern
   • Elasticsearch details | Time field name: Opened
   • Elasticsearch details | Min time interval: 5m

The data source page should look something like this:

...

Querying with Elasticsearch

Now that we have a functioning data source, let’s build a regular time series visualization panel to get a feel for how we’re going to present the data. Remember that we want to be able to look at the kinds of calls made to 311 across different neighborhoods, so let’s first make a query just to get an idea of how many graffiti calls are made. You can do this in Explore, but we’re going to build on our panel as we go, so you’ll want to start with a dashboard and a new time series panel visualization.

Go to the panel’s Query tab to access the Elasticsearch data source query. You’ll notice its similarity to the InfluxDB Query tab from previous chapters. The terminology may be a little different between Elasticsearch and InfluxDB, but the concepts are very similar. Let’s enter a query string into the Query field. Elasticsearch leverages a powerful Google-like search engine called Lucene to perform text...

Creating a template variable

Now that we have an idea of what our panel might look like, we observe that if we were to create additional panels, one for each category of 311 call, we might need to create many panels, with each being a virtually identical panel, except for the value of the category in question. This seems tedious at best, and a potential maintenance nightmare at worst. What would happen if, say, you wanted to tailor the color scheme for the breakdowns or change the stacking options? You might end up clicking through dozens of panels, just to make a single change.

What if we could put a placeholder string in our query, and then set the value of that placeholder from a menu selection? You would then be creating a kind of template panel, with a placeholder standing in for the actual query value. In the panel query, you would fill a template panel variable. A template variable is a special string that represents a placeholder that Grafana will substitute from a selection...

Creating a new dashboard

Now, we have everything we need to begin building some dashboards with repeating rows and panels. Observe that the data fields group roughly into four major types:

• Call type: (Category and RequestType)
• Responsibility: (PoliceDistrict, ResponsibleAgency, and SupervisorDistrict)
• Location: (Address, Neighborhood, Street, Latitude, and Longitude)
• Date: (Opened, Closed, and @timestamp)

What we will do here is create a series of dashboards, each one dedicated to a Responsibility entity. Each dashboard will be split into two-column panels, one for each Call type. We will then allow the user to decide the Group By terms for the panel queries. We’ll need to create several template variables to drive the dashboard, but once they’re created, things should fall into place. Let’s get started!

First, we will need to create a dashboard. On the dashboard, we’ll create some template variables.

Setting up the template...

Linking dashboards

Now that you have your dashboards set up, you may have noticed that navigating between dashboards can be a bit tedious. To go to a different dashboard, you click the dashboard’s name, click the Grafana logo, or click the Dashboards sidebar menu, and then you look for your dashboard and click on it. This isn’t very efficient, and it makes it difficult to deploy your dashboards as a coherent site that doesn’t force your users to go rummaging through a lot of dashboards that aren’t relevant to them, in search of the ones that are.

Fortunately, you’re in luck! Grafana provides a simple, dashboard-level linking system to facilitate the creation of navigable dashboards. Dashboard linking supports intra-dashboard links via tagging, or inter-dashboard via URL. Let’s see how that works for our newly created dashboards.

Adding dashboard tags

The first step is to make a copy of the dashboard you created in the previous section...

Annotating dashboards

Annotations are a versatile mechanism to highlight individual events in a time series. By singling out a single data point at a particular time and marking it with metadata, you have the capability to mark up your dashboard panels with rich data, such as text and tagging. Grafana provides two annotation capabilities to choose from – native annotation queries are created interactively and stored with the dashboard on the Grafana server, while data source annotation queries are created as data source queries, with the query and annotation configuration stored with the dashboard.

Annotating the graph panel

Since data source annotations can be resource-intensive, we’ll demonstrate annotation with just a single-panel dashboard. Create a new dashboard and a single Time series panel with the following settings:

• Data Source: Elasticsearch
• Query: RequestType:Graffiti
• Query options | Min interval: 1h
• Legend | Visibility: on
• Graph...

Sharing dashboards

Now that you’ve created these lovely dashboards, how do you share them with the world (or just your boss)? One of the first questions you must consider is the link between your dashboards and your data sources. For the sake of this exercise, we’ll keep the data sources and the dashboards in proximity – that is, on the same host and the same network. If you wish to share your dashboards, they will need to access your machine and possibly your network. You’ll want to give some serious thought to how best to share them. Here, we’ll discuss a few strategies and their pros and cons.

Sharing dashboard links

The most straightforward sharing mechanism is to simply give out a URL to your Grafana server that references the dashboard in question. Click the Share dashboard button (the three circle interface icon in the dashboard bar and use the Link tab to create a link. If you want the dashboard to preserve the current view, make sure...

Summary

We’ve accomplished a lot in this chapter. Here, we created an Elasticsearch server, imported a realistic textual dataset with Logstash, and learned about several different types of template variables, as well as how to set them up to parameterize our dashboards. Then, we applied template variables to repeat rows and panels and discovered the native annotation feature, as well as how to create annotations from an Elasticsearch query. Finally, we explored different sharing options and their pros and cons.

At this point, you have been exposed to enough of Grafana to now go off and create dashboards built around your own data sources. We’ve looked at two of the more popular data sources, InfluxDB and Elasticsearch, but we’ve only scratched the surface of their capabilities, apart from being used with Grafana. I encourage you to explore them; the more you understand how data sources manage data, the better you will be able to tailor your datasets to get the...