Preface

Grafana is an open source analytical platform used for analyzing and monitoring time-series data. This second edition will help you to get up to speed with Grafana 10’s latest features for querying, visualizing, and exploring logs and metrics no matter where they are stored. Along the way, we’ll introduce key concepts and best practices in software development, data visualization, and application administration.

The book begins by showing you how to quickly install and set up a Grafana server using Docker. You’ll become familiar with important components of the Grafana interface and learn how to analyze and visualize data from sources including InfluxDB, Telegraf, Prometheus, Logstash, and Elasticsearch.

We will cover many of Grafana’s key panel visualizations, including Time Series, Stat, Table, Bar Gauge, and Text. You’ll use Python to pipeline data, transformations to facilitate analytics, and templates to build dynamic dashboards. You’ll explore real-time data streaming with Telegraf, Promtail, and Loki, as well as observability features such as alerting rules, PagerDuty, and Slack integrations.

As you progress, the book will delve into the administrative aspects of Grafana, from configuring users and organizations to implementing user authentication with Okta and LDAP, organizing dashboards into folders, and more.

By the end of this book, you’ll have gained the knowledge you need to extract, transform, and load data; connect Grafana to time-series databases; build interactive dashboards; and leverage ad hoc data exploration for observability. Whether you are interested in visualization for data science or observability for your operations, this book will provide the launch pad for anyone looking to become proficient using a data visualization and observability application such as Grafana.

Who this book is for

This book is for business intelligence developers, business analysts, data analysts, and anyone interested in performing time-series data analysis and monitoring using Grafana. Those looking to create and share interactive dashboards or looking to get up to speed with the latest features of Grafana will also find this book useful. Although no prior knowledge of Grafana is required, basic knowledge of data visualization and some Python programming experience will help you understand the concepts covered in the book.

What this book covers

Chapter 1, Introducing Data Visualization with Grafana, provides a brief introduction to the use of data visualization in general and specifically in Grafana. We will then move on to installing a Grafana server onto your machine, using either a native installer or a Docker container. Launching the server and connecting to it with a web browser will also be covered.

Chapter 2, Touring the Grafana Interface, will explore the workings of the major UI components after you have launched and connected to the Grafana web application. We will look at the search bar, side menu, and Home dashboard.

Chapter 3, Diving into Grafana's Time Series Visualization, will dive into the Time series panel visualization for a closer look at how to work with the major components of the main Grafana visualization. After connecting to a test data source, we will also identify common panel components in preparation for working with other visualizations.

Chapter 4, Connecting Grafana to a Prometheus Data Source, will show you how to launch the Prometheus time-series database from a Docker container, load an actual time-series dataset, and query and visualize data in Grafana.

Chapter 5, Extracting and Visualizing Data with InfluxDB and Grafana, will show how to write a simple Python Extract, Transform, and Load (ETL) script to access data from a public data server and push it to InfluxDB. We’ll also connect Grafana to InfluxDB and try out some more advanced query techniques.

Chapter 6, Shaping Data with Grafana Transformations, will introduce the concept of the Grafana data frame, and how the different Grafana transformations can shape query data. We’ll also chain transformations into a more complex data pipeline.

Chapter 7, Surveying Key Grafana Visualizations, will see us use the Table, Stat, Bar Gauge, and Gauge panel visualizations to display our weather data.

Chapter 8, Surveying Additional Grafana Visualizations, will see us modify the Python ETL script to download earthquake data. We’ll visualize the data using Geomap, Bar chart, Histogram, and Heatmap visualizations.

Chapter 9, Creating Insightful Dashboards, uses what we’ll have learned about Grafana panel visualizations and some basic information design principles to create production dashboards for visualizing weather and earthquake data.

Chapter 10, Working with Advanced Dashboard Features and Elasticsearch, explores the powerful advanced features of the dashboard, including annotations, templating with variables, and dashboard linking, as well as techniques for sharing dashboards. We’ll pull down public data from the city of San Francisco and use Logstash and Elasticsearch as the data source.

Chapter 11, Streaming Real-Time IoT Data from Telegraf Agent to Grafana Live, will present the first in a trilogy of chapters on observability by introducing the concept of real-time data streaming. We’ll build a data pipeline to stream data from an Internet of Things (IoT) simulator using standard MQTT protocols and then use InfluxDB to send the messages to Grafana Explore.

Chapter 12, Monitoring Data Streams with Grafana Alerts, will show you how to take streaming data, monitor it for anomalies with alerting rules, and connect those alerts to a set of notification channels, including email, PagerDuty, and Slack.

Chapter 13, Exploring Log Data with Grafana’s Loki, will complete the observability trilogy by showing how to capture observability metrics and logs with the combination of Promtail and Loki. We’ll perform an ad hoc analysis with Explore to check for correlations between metrics patterns and logging events.

Chapter 14, Organizing Dashboards and Folders, will show you how to label dashboards and organize them into folders to make them easier to find. We’ll also look at other dashboard features, such as starred dashboards, dashboard playlists, and the Dashboard list panel visualization.

Chapter 15, Managing Permissions for Users, Teams, and Organizations, will show you how to manage users, teams, and organizations, including access control and user addition and deletion.

Chapter 16, Authenticating Grafana Logins Using LDAP or OAuth 2 Providers, will show you how managers can connect Grafana user authentication to a variety of services. We’ll authenticate using an internal LDAP server and use OAuth 2 to authenticate using external services from GitHub, Google, and Okta.

Chapter 17, Cloud Monitoring AWS, Azure, and GCP, will show how Grafana can provide monitoring support for a variety of services provided by major cloud platforms, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).

To get the most out of this book

In order to complete the majority of the exercises in this book, you will need to download and install Docker along with Docker Compose. For the examples in the book, we will be downloading and installing other software and datasets, including Grafana and Loki, so you will occasionally need an internet connection. You can download and install each software package independently, but our tutorial instructions are designed to work with Docker. We do that so that all software dependencies and network management can be encapsulated within the Docker container paradigm.

We will run a fair amount of software from the command line, so you should be comfortable with typing commands into a shell, such as Bash or Windows PowerShell. To access the contents of the book’s GitHub repository, you will either need Git or an unzip application.

Having an interest in science in general and data science, in particular, will go a long way toward making this book interesting and useful. It would also be helpful to have some programming experience with a scripting language such as Python, but since all the code is included, you can run it directly from a clone of the book’s GitHub repository. Some familiarity with relational databases will help you understand some of the terminology and concepts behind time-series databases.

Software/hardware covered in the book	Operating system requirements
Grafana	Windows, macOS, or Linux
Docker	Windows, macOS, or Linux
Loki/Promtail	Windows, macOS, or Linux
Prometheus	Windows, macOS, or Linux
InfluxDB/Telegraf	Windows, macOS, or Linux
Elasticsearch/Logstash	Windows, macOS, or Linux
OpenLDAP	Windows, macOS, or Linux
Python 3.7+	Windows, macOS, or Linux

Grafana is an application under constant development and revision, and as such, the depictions, descriptions, and illustrations in this book represent a snapshot in time and are current at the time of writing. By the time you read this book, features may have been added, altered, or deleted outside of our control. However, we believe any deviations from the book should be easily accommodated with only minor adjustments.

It might also be helpful to use an IDE application such as Microsoft Visual Studio Code, or JetBrains PyCharm.

In order to follow along with the exercises in Chapter 16, Authenticating Grafana Logins Using LDAP or OAuth 2 Providers, you will need accounts with GitHub, Google, and Okta. To follow the exercises in Chapter 17, Cloud Monitoring AWS, Azure, and GCP, you will need to create an account with AWS, GCP, and Microsoft Azure.

The examples and software in this book have not been validated for security reasons. They require an external internet connection and leverage open source software under a variety of licenses, so if you intend to use any of this software within a security-conscious computing environment (such as in an education or corporate environment), it is highly recommended that you consult your local IT professionals in advance.

I hope to show with the examples in this book how easy it is to build simple data visualization pipelines with Grafana and today’s open source tools. I also hope this book will inspire and empower you to seek out your own datasets to acquire, analyze, and visualize. Best of luck!

If you are using the digital version of this book, we advise you to type the code in yourself or access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code. Each chapter folder includes dashboards, docker-compose.yml files, and a Makefile to help out when running some of the command-line tools.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Learn-Grafana-10. Any update to the code will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system.”

A block of code is set as follows:

FROM python:3
SELECT mean("value") FROM "temperature"
WHERE $timeFilter
GROUP BY time($__interval), "station" fill(none)

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

listener 1883
allow_anonymous true

Any command-line input or output is written as follows:

% docker-compose down

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “Click on Load Data | API Tokens.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [e-mail] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Learn Grafana 10.x, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

1. Scan the QR code or visit the link below

https://packt.link/free-ebook/9781803231082

2. Submit your proof of purchase
3. That’s it! We’ll send your free PDF and other benefits to your email directly