Chapter 12: Using Grafana with OpenSearch

In this chapter, you will learn about a powerful analytic platform – OpenSearch – and how to integrate it with Grafana.

With OpenSearch and Grafana, you can bring to your IoT project advanced analytics and impressive visualizations.

Across this chapter, you will learn the following:

• What OpenSearch is and what it is used for
• How to install and configure OpenSearch
• How to ingest data into OpenSearch
• How to integrate OpenSearch and Grafana
• How to visualize metrics and alerts coming from OpenSearch in Grafana

Without a doubt, you can use OpenSearch and Grafana together to build a powerful analytic platform.

Technical requirements

To take advantage of this chapter, you will need the following:

• A Grafana instance running
• An OpenSearch instance running – we will see how to install it
• Data coming from sensors or systems
• Data-ingesting software

The best option for trying these platforms is deploying them using Docker, which completes the work in minutes.

Let’s see what OpenSearch is and what it is used for.

What OpenSearch is and what it is used for

OpenSearch is a fork from the Elasticsearch project. It happened after Elasticsearch became a non-open source project, early in 2021. The first version of OpenSearch appeared in the middle of the year 2021.

The major difference between the two projects is the advanced analytics capabilities, which are available for free in the OpenSearch version, whereas you have to pay for their use in ElasticSearch.

The OpenSearch system allows searching and analyzing data from several sources. You can build visualizations, run alarms, and create analytics. With OpenSearch, you can also implement anomaly detection and data classification.

In other words, OpenSearch offers very powerful tools for analyzing data. In this chapter, you will learn how to ingest data into OpenSearch, perform analytics on it, and show it in Grafana dashboards.

You can use OpenSearch for many things, such as the following:

• Data store and search engine
• Log...

Installing OpenSearch

At the time of writing, there are three methods for installing OpenSearch – Docker, tarball, and an Ansible recipe. Here, you will learn how to install it using Docker because that is the easiest and fastest way to deploy an OpenSearch instance, or even a cluster.

You can reach the Docker images in the Docker hub through the following link: https://hub.docker.com/u/opensearchproject.

To get the latest images of OpenSearch and OpenSearch Dashboards, you can run the following commands:

docker pull opensearchproject/opensearch:latest

docker pull opensearchproject/opensearch-dashboards:latest

To deploy a single instance of OpenSearch, run the following command:

docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchproject/opensearch:latest

To check that it’s working, go to your web browser and point it to this address: https://localhost:9200.

Accept the security risk warning of your browser,...

Ingesting data in OpenSearch

In this section, we will work around the system described in Figure 12.1. There, you can see data-ingesting components on the left of OpenSearch. On the other side, OpenSearch feeds Grafana dashboards:

Figure 12.1 – System architecture

You can ingest data in OpenSearch by sending PUT or POST HTTP requests. So, you can send data to OpenSearch from a script, using curl or some similar command.

In this section, you will learn how to implement a data ingestion pipeline for use in IoT systems. In this case, we will use Fluent Bit. Let’s see how to do it.

Fluent Bit

Fluent Bit is an open source log processor. It lets you gather data, process it, and send it to other systems, such as OpenSearch.

You can reach this project on Github at the following link: https://github.com/fluent/fluent-bit.

You can install Fluent Bit on several Linux distributions, several versions of Windows, and macOS. To see detailed instructions...

Integrating OpenSearch and Grafana

Grafana allows you to get OpenSearch data using a data source plugin. You can see the plugin at the following link: https://grafana.com/grafana/plugins/grafana-opensearch-datasource/.

To install this plugin, you can follow any of the procedures discussed in Chapter 7, Managing Plugins.

Adding the data source

Once you have installed the plugin, you can add a new data source from the data source configuration page.

Important Note

The OpenSearch plugin can't be installed on the build of Grafana made for Raspberry Pi.

To connect Grafana to the OpenSearch database, you’ll have to configure the plugin. Let’s see the configuration options.

First, you have to set the basic parameters:

• Name: Here, you can name the data source. This is the name that appears when you use this data source in the dashboard panels.
• Default: Select this if you want to set this data source as the default.
• URL: The full URL...

Summary

In this chapter, you have learned how to use several tools for ingesting, storing, and analyzing data. These tools are Fluent Bit, OpenSearch, and OpenSearch Dashboards.

You also saw how to integrate Grafana with OpenSearch using the data source plugin.

You additionally learned how to build dashboards in Grafana using the OpenSearch data source and the query interface.

OpenSearch offers a lot of functionalities and advanced analytics. You may want to explore these options in more detail.

In the next chapter, you will learn how to integrate a useful network management system – LibreNMS – with Grafana.

Invitation to join us on Discord

Read this book alongside other Grafana users and the author Rodrigo Juan Hernández.

Ask questions, provide solutions to other readers, chat with the author via Ask Me Anything sessions and much more.

SCAN the QR code or visit the link to join the community.

https://packt.link/iotgrafana