Connecting Grafana to a Data Source

In previous chapters, we took a whirlwind tour of the Grafana UI. We looked at how graph panels query for datasets via data sources and how panels combine to form dashboard pages.

In this chapter, we will begin to apply our newly gained skills to more practical considerations. We will use real data, where possible, for analysis with a focus on solving real-world scenarios and create the kinds of comprehensive dashboards you would expect to see in a production environment.

Our first step in this journey begins (as always) with data. Here, we will configure a live database serving actual web service data (generated by Grafana itself!). We'll pull that data into Grafana as a data source, then we'll use the Explore tool to get a feel for what kinds of metrics are available. We'll also look at visualizing the data through a variety of queries. Finally, we'll learn...

Technical requirements

The tutorial code, dashboards, and other helpful files for this chapter can be found in the book's GitHub repository at https://github.com/PacktPublishing/Learn-Grafana-7.0/tree/master/Chapter04.

Installing the Prometheus server

Our first task is to get the Prometheus server up and running so that we can start serving real data. Prometheus is a powerful open source time-series database and monitoring system originally developed by SoundCloud. It followed Kubernetes to become the second Cloud Native Computing Foundation graduating incubation project. Grafana, having partnered with the Prometheus maintainers, includes the Prometheus data source as a first-class data source plugin.

Tutorial code, dashboards, and other helpful files for this chapter can be found in this book's GitHub repository at https://github.com/PacktPublishing/Learn-Grafana-7.0/tree/master/Chapter04.

Installing Prometheus from Docker

We're going to start up Prometheus from Docker Compose and point it to a local configuration file. First, let's create the following configuration file and save it to our local ch4/prometheus directory as prometheus.yml:

global...

Exploring Prometheus

Once we have the Prometheus data source properly configured, you might be wondering what kind of data we're likely to see. Turns out, since we configured Prometheus to scrape itself, we'll get a bunch of juicy internal server metrics delivered to the scraped endpoint and stored in the Prometheus database. So, let's dive in and get an idea of what's there.

Using Explore for investigation

Selecting Explore from the left-hand side menu activates the Explore tool. Basically, Explore includes special versions of both Graph and Table panel plugins, each looking at the same data source query. Make sure to select your Prometheus data source from the dropdown, then select a metric data series by selecting up from the Metrics menu. This is probably the simplest metric available: it shows 1 if the server is up and 0 otherwise. You can see, from the following screenshot, that (obviously) our Prometheus server is up and running:

...

Querying the Prometheus data source

Now that we have both Prometheus and Grafana logging metrics, let's try out some queries. I won't be able to give you a full rundown on every aspect of PromQL—the Prometheus query language—but I can help you to learn enough to be able to examine many of the server metrics that can be accessed from the Prometheus data source.

To get a better understanding of how queries work in time-series databases such as Prometheus, let's first start with a more traditional database, such as MySQL. Typically, the structure of a query looks something like this:

SELECT some fields
 FROM some table
 WHERE fields match some criteria

You get back from the query some rows, each one containing the contents of some fields. In the case of time-series databases, things work a little differently. The query has a form that is more like the following:

SELECT metric
 FROM some data store
 WHERE metric tags match some...

Detecting trends with aggregations

As we continue up the stack, let's now take a look at some server performance metrics. How about an obvious web server metric? Enter prometheus_http_requests_total to get an idea of how many requests have been served so far:

Well, this is a bit of a mess. You can't see all 22 of the time series—they're all stacked on top of each other and there's a vague warning about something monotonically increasing. As we saw in the previous section, it's no problem to apply filters—say, to filter out the GET method handlers—but then we'd still have a stack of nearly 20 individual series.

Applying aggregations to our query data

If only there were some way to combine all the individual data series into one. It turns out there is, and it's called an aggregation. We can actually tell Prometheus to apply an aggregation function (in this case, sum) after...

Understanding the data source limitations

After seeing how powerful even relatively simple PromQL queries can be, it is tempting to think you can query and graph virtually any metric in your data source. Unfortunately, there are limitations to certain kinds of calculations, either imposed by the nature of the data or by the data source application.

It is important to remember that when you create a graph, you are entering into a trust relationship between you and your audience (which might even be you). When you place a pixel on a graph that isn't explicitly represented by a corresponding data point, you are asking your audience to accept that what you are doing is, in essence, reconstructing a signal from the underlying data.

Therefore, you have an obligation to respect the integrity of the data and not abuse that trust by manipulating the data to say things that aren't true or lead the viewer to draw erroneous conclusions.

Throughout this book...

Summary

We've come to the end of our introduction to data sources and how to create queries for them. As a treat, go to Configuration | Data sources and edit your Prometheus data source configuration. You'll notice a second tab called Dashboards. Under that tab is a set of dashboards curated to work with a Prometheus data source.

If you import the Grafana metrics dashboard, you'll now have a full dashboard with a number of Graph and Singlestat panels. Some of the queries we tried out in this chapter were inspired by those dashboard panels. Open and edit them to get a look at the queries, see how they were constructed, and note the techniques that were used to extract information from the data series for use in the legend.

In fact, one of the best ways to learn a complex application such as Grafana is to simply crack open the dashboard panels from different sources and glean knowledge from the work of others. One of the more pleasant aspects of Grafana...