Chapter 7. Visualizing data with Kibana

Kibana is an open source web-based analytics and visualization tool that lets you visualize the data stored in Elasticsearch using a variety of tables, maps, and charts. Using its simple interface, users can easily explore large volumes of data stored in Elasticsearch and perform advanced analysis of data in real time.

In this tutorial, let's explore the various components of Kibana and explore how one can use it for data analysis. 

We will cover the following topics in this chapter:

• Downloading and installing Kibana
• Data discovery using Kibana
• Visualizations in Kibana
• Analysis of time-series data with Kibana
• Configuring and developing well known plugins in Kibana 

Downloading and installing Kibana

Just like with other components of Elastic Stack, downloading and installing Kibana is pretty simple and straightforward.

Navigate to https://www.elastic.co/downloads/kibana#ga-release and, depending on your operating system, download the ZIP/TAR file as shown in the following screenshot: 

Note

The Elastic developer community is quite vibrant, and newer releases with new features/fixes get released quite often. While you have been reading this book, the latest Kibana version might have changed. The instructions in this book are based on Kibana version 6.0.0. You can click on the past releases link and download version 6.0.0 if you want to follow as is, but the instructions/explanations in this book should hold good for any 6.x release.

Kibana is a visualization tool that relies on Elasticsearch for querying data that is used for generating visualizations. Hence, before proceeding further, make sure Elasticsearch is up and running.

Installing on Windows

Unzip the...

Data preparation

As Kibana is all about gaining insight from data, let's load some sample data that we will use as we follow the tutorial. One of the most common use cases is log analysis. For this tutorial, we will be loading Apache server logs into Elasticsearch using Logstash and then using it in Kibana for analysis/building visualizations.

https://github.com/elastic/elk-index-size-tests hosts a dump of Apache server logs that were collected for the site www.logstash.net for the period of May 2014 to June 2014. It contains 300,000 log events.

Navigate to https://github.com/elastic/elk-index-size-tests/blob/master/logs.gz and click the Download button. Unzip the logs.gz file.

Make sure you have Logstash version 5.6 and above installed. Create a config file named apache.conf in the $LOGSTASH_HOME\bin folder, as shown in the following code block:

input 
{ 
  file {
        path => "D:\Learnings\data\logs\logs"
        type => "logs"
        start_position => "beginning"
       }
}
...

Kibana UI

Open up Kibana from the browser using the http://localhost:5601 URL. The landing page will look as follows:

User interaction

Let's understand the user interaction before diving into the core components of Kibana. A typical user interaction flow is as depicted in the following diagram:

The following points will give you a clear idea of user interaction flow in Kibana:

• Prior to using Kibana for data analysis, the user would have already loaded the data in Elasticsearch.
• In order to analyze the data using Kibana, the user has to first make Kibana aware of the data stored in ES indexes. So the user would need to configure the indexes on which he wants to perform analysis.
• Once configured, the user has to find out the data structure, such as the fields present in the document and the type of fields present in the document, and explore the data. This is done so that he can decide how he can visualize this, and what type of questions he wants to pose and find answers for in terms of the data...

Timelion

Timelion is a visualization tool for analyzing time-series data in Kibana. It enables you to combine totally independent data sources within the same visualization. Using its simple expression language, you can execute advanced mathematical calculations, such as dividing and subtracting metrics, calculating derivatives and moving averages, and visualizing the results of these calculations.

Timelion UI

Timelion is present in the left pane of the Kibana UI, between the Dashboard and Dev Tools icons:

The main component/feature of the Timelion UI is the Timelion query bar that allows one to define expressions that influence the generation of the graphs. It allows one to define multiple expressions separated by commas, and also allows you to chain functions.

The Timelion UI also offers the following options:

• New: This is used to create a new Timelion sheet for creating graphs.
• Add: One can create multiple charts on the same same Timelion sheet using this option.
• Save: This is used to save the...

Using plugins

Plugins are a way to enhance the functionality of Kibana. All the plugins that are installed will be placed under the $KIBANA_HOME/plugins folder. Elastic, the company behind Kibana, provides many plugins that can be installed, and there are quite a number of public plugins that are not maintained by Elastic which can be installed, too.

Installing plugins

Navigate to KIBANA_HOME and execute the install command, as shown in the following code, to install any plugins. During installation, either the name of the plugin can be given (if it's hosted by Elastic itself), or the URL of the location where the plugin is hosted can be given:

$ KIBANA_HOME>bin/kibana-plugin install <package name or URL>

For example, to install x-pack, a plugin developed and maintained by Elastic, execute the following command:

$ KIBANA_HOME>bin/kibana-plugin install x-pack

To install a public plugin, for example LogTrail (https://github.com/sivasamyk/logtrail), execute the following command:

$ KIBANA_HOME...

Summary

In this chapter, we covered how to effectively use Kibana to build beautiful dashboards for effective storytelling about your data.

We learned how to configure Kibana to visualize data from Elasticsearch. We also looked at how to add custom plugins to Kibana.

In the next chapter, we will cover ElasticSearch, and the core components that help when building data pipelines. We will also cover visualizing data to add the extensions needed for specific use cases.