Chapter 4. Kibana Interface

After exploring Elasticsearch and Logstash, it's time for Kibana, which provides us with an interface to visualize the data we collect and store. In this chapter, we will focus on the Kibana interface, providing all the important details. We will also learn about Lucene query searches, aggregations, and so on, and understand the role they play.

At the end of this chapter, you will understand the new features of Kibana, the functionality of various tabs and their components along with how to create visualizations, dashboards, and customize Kibana settings. Also you will understand how to insert data into Elasticsearch, create an index pattern, and visualize the data, followed by how to create dashboards and share and embed them.

In this chapter, we will cover the following sections:

• Kibana and its offerings

• Exploring the discover interface

• Querying and searching data

• Exploring the visualize interface

• Understanding aggregations

• Exploring the dashboard interface

• Understanding...

Kibana and its offerings

Kibana is a visualization tool that is used for visualizing data (structured or non-structured) stored in Elasticsearch. Kibana acts an over-the-top layer of Elasticsearch which is used to search, view, analyze data, and create visualizations on the data that is stored in Elasticsearch indices. Kibana and Elasticsearch are Elastic products, but both use the Apache license, which uses the powerful capabilities of Elasticsearch (Lucene query syntax, aggregations) and is based on Apache Lucene. Kibana provides the ability to do analysis of data, visualize the data in forms of different charts such as the Area Chart, Line Chart, Bar Chart, Pie Chart, Tile map,Metrics, and Data table, and create dashboards that can even be shared or embedded. Dashboards provide a unified way to showcase all your visualizations in one place. All the searches/visualization/dashboard are stored as JSON and the underlying has JSON documents. The JSON files for searches/visualization/dashboards...

Exploring the discover interface

The Discover page helps you to play around with the data by easily analyzing the indexed documents. It allows you to perform different types of searches on the data, helping you to understand what the data means or how the data can be used to create visualizations. It provides the ability to choose different index names by changing the index pattern without leaving the Discover page. You can easily perform search queries, use filters, and view documents matching the queries and filters.

A Discover page interface typically looks like the following:

The Discover page uses the following components:

• Time Filter: This filters the data for a specific time range

• Search Box: This is used to search and query the data

• Toolbar: This contains options such as new search, save search, open saved search, and share

• Index Name: This displays the name of the selected index

• Fields List: This displays the name of all the fields that are present within the selected index

• Number...

Time Filter

This is used to drill down the data based on time. It provides the flexibility to search/query/filter the data on any time period as required. It makes it easier to analyze data of a specific time period or customize the time period as per the requirement. It also provides various options for setting the Time Filter.

Note

By default, the Time Filter is set as the last 15 minutes, which means it displays data of the previous 15 minutes only.

Upon clicking on the Time Filter, we can find various sub-options as mentioned in the following sections.

Quick time filter

This provides numerous time range options to quickly use the Time Filter for viewing data such as today, this week, year to date, this day last week, previous month, last 30 minutes, last 24 hours, last 90 days, last 5 years, and so on.

Relative time filter

It is used to filter the data based on the relative time with respect to the current time. It provides a From and a To box to specify the relative time from Now. The various...

Querying and Searching data

The search box is used to perform various types of queries that fetch the matching documents. Upon searching, the whole Discover page along with its components gets automatically refreshed. Kibana uses the underlying powerful capabilities of Lucene query syntax using the data that is queried. As Kibana utilizes the functionality of the underlying Elasticsearch, Lucene queries provide the ability to perform various types of searches ranging from simple to complex queries.

Lucene queries provide a number of ways to search data. Let's look at these one by one.

Full-text searches

This is used to search for a term within the complete text. The different ways to search for full-text are shown as follows:

• Search for the single term:

  Example: To search for a phrase, type elasticsearch kibana in the search bar.

• Search for a phrase (group of words):

  Example: To search for a single term, type kibana in the search bar.

  Note

  By default, phrases use, OR Boolean operator to search for...

Fields and filters

The field list is used to provide the list of all the fields that are used in the data. It is useful in analyzing the dataset to learn what the fields are and what type of information the field contains. Fields are categorized under selected fields and available fields wherein the name of each field is arranged in alphabetical order.

It also provides the option of knowing about the data present in the field, the top five values of the field, and its percentage breakdown for the documents containing the value. Such information is available by clicking on any of the fields from the Fields List. Also you can add the fields to view the data of that field for all the documents by clicking on Add, which is present beside the field name. It will move the field name under selected fields, which will provide you with the information of the values present in the field.

Filtering the field

This is used to filter the search results on the basis of the values of the field. Field searches...

Discovery page options

In the toolbar along with the search bar, there are other options, such as:

• New Search: This is used to create a new search query by erasing any present search query.

• Save Search: This is used to save the search query along with the selected index.

• Open Saved Search: This is used to load the existing saved search queries with the specified index. It will open the saved search query and the index with which it was saved.

• Share: This is used to share the search query via a link for which a short URL can also be generated or shared directly. It can also be used to share the saved search.

• Adding fields to the Document Data: Fields can be easily added to the document data by clicking the Add button, which is present beside the field name under the fields list upon hovering of the mouse cursor.

• Removing fields from the Document Data: Fields can be easily removed from the document data by clicking the Remove button, which is present beside the field name under the Fields...

Exploring the visualize interface

Visualization is the heart of Kibana and it is one of the sole reasons behind the creation of Kibana to provide users the capability to visualize large volumes of data. As the adoption of Elasticsearch and Logstash was increasing day by day, the ability to directly visualize data stored in Elasticsearch was missing and users were unable to make sense of the data. Then came the savior, Kibana, which solved the problem and provided a simple yet intuitive interface. It solved the challenge of visualizing huge volumes of data in near real time. It is the core component that makes Kibana a functionally rich open source software.

The Visualize page helps to visualize all the data that has been stored in Elasticsearch. After understanding the data using the Discover page, the Visualize page takes it a step further and provides the ability to build visualizations with ease. Visualizations help to understand the data instead of going through tons of raw data, which...

Exploring the Dashboard interface

Dashboard provides a unified view of displaying all your visualization in one place. Dashboard provides a collection of multiple visualizations or searches that can be arranged in any way and it allows the ability to resize, move, edit, or remove any visualization added to the dashboard. The dashboard provides real time insights to the streaming data as all visualizations are updated in the dashboard in real time. Updating the visualization reflects instantly across the dashboards using that visualization. Dashboard also provides the ability to use search queries that update the visualizations present in the dashboard as per the search result.

Note

For creating a dashboard, visualizations/searches need to be saved.

The Dashboard page interface typically looks like the following screenshot:

The Dashboard interface is fairly simple to understand as it displays the main Kibana header, which is the same across all pages in Kibana that contain the Time Filter.

The...

Understanding Timelion

Timelion is the latest addition to the Kibana UI with the introduction of Elastic Stack. Timelion is used for analysis and visualizations of time-series data. It provides the ability to combine multiple data sources into a single visualization and gives a range of mathematical calculations that can be used, such as cumulative sum, derivative, moving averages, and so on.

Timelion is present in the left-pane of the Kibana UI between icons of Dashboard and Dev Tools. It has its own language and expressions when using it, which makes it difficult to start with. However, it has a great built-in documentation and tutorial to guide you on how to start using Timelion.

When you click on Timelion, you will be greeted with a screen similar to the following screenshot:

In the preceding screenshot, you see that there has been a default expression that has already been added, .es(*),which means it is querying all the data present in Elasticsearch within all the indices. It represents...

Exploring Dev Tools

Dev Tools refers to the development tools that aid the developer. In Kibana, it is used for the Console UI, which provides a simple yet clean interface to access API queries using the REST API exposed by the Elasticsearch client. Console allows us to make any API call from a web browser. Its interface gives us a clean way to make a call and generates JSON in a pretty print format, which allows you to view results in a neat way. It works on top of an HTTP layer of an Elasticsearch cluster.

Upon clicking Dev Tools, you will be greeted with the Console UI, as shown in the following screenshot:

For a better understanding all of the available options are marked in the screenshot from number 1 to 7, which are described this section:

1. Editor Pane: This is the area where we write our request. It uses commands in cURL similar syntax and presents it in a simpler way. For example, by default, a query displayed in Console is:

           GET _search 
           { 
             "query...

Exploring the Management interface

Settings provide a way to customize and tweak various Kibana related properties.

This page has been categorized into multiple options to understand the various settings involved with each of the options. The options are displayed as shown in the following screenshot:

Let us have a closer look at the various settings provided by each of the options.

Index patterns

This option provides us with the ability to configure an index or indices pattern to be used with Kibana. It provides the various indices patterns that have already been configured to use with Kibana along with the ability to view the information related to various fields present in an index.

You can configure various types of index, such as:

• Index Name

• Index Patterns

Index patterns are divided into the following two search categories:

• Wildcard Searches: logstash-* will fetch all indices names starting with logstash-

• Event Time Based: [logstash-]YYYY-MM-DD will fetch all indices following a pattern...

Putting it all together

After learning about Elasticsearch, Logstash, and Kibana, let's use these components to create an end-to-end pipeline to parse data from Logstash to Elasticsearch and visualize it using Kibana. We will use a CSV file as input data, which will be used to analyze and create visualizations out of the data.

This will help us to quickly get started by using all the three components together to create an end-to-end pipeline. While using this example in this chapter, we assume that you have successfully installed Elasticsearch, Logstash, and Kibana as described in Chapter 1, Elastic Stack Overview.

Let's have a look at the input data.

Input data

We will be using the input data provided by United States Department of Agriculture Economic Research Service (USDA ERS), which is about educational attainment for the U.S and counties, 1970-2014. It provides information about the attainment of education, people who have less than a high school diploma, a high school diploma only, those...

Summary

In this chapter, we learned about Kibana, its interfaces, and got a glimpse of what it is capable of. Using examples, we created several types of visualizations to analyze the data. Kibana is becoming very popular among analytics tools these days for its offerings. We will be learning more in the following chapters where we will see the real-time datasets, log analysis on production servers, and so on.

In the next chapter, we will learn about Beats, which is a new component of Elastic Stack. We will see the types of Beats, their usage, and the role they play in Elastic Stack with examples.