Chapter 8: Interacting with Your Data on Kibana

As we've explored in the previous chapters, Elasticsearch is a powerful and versatile tool to store, query, and aggregate data. The only way to interact with Elasticsearch is by using its feature-rich set of REST APIs. This includes anything from creating and managing indices and ingesting documents to running queries or aggregating large datasets. We've also looked at how tools such as Beats and Logstash are great at collecting data from various sources and loading it into Elasticsearch clusters for end user consumption. This is where Kibana plays a vital role in the Elastic Stack.

This chapter explores the role that Kibana plays in the Elastic Stack in allowing users to visualize, interact with, and build use cases on top of data in Elasticsearch. Kibana is also the primary way in which users can consume out-of-the-box solutions, such as Enterprise Search, Security, and Observability, as well as manage and configure the...

Technical requirements

This chapter walks you through the various features of Kibana when it comes to building and consuming use cases from your data. You will need access to an instance of Kibana connected to an Elasticsearch deployment to follow along. If you don't already have a deployment configured, follow the instructions provided in Chapter 2, Installing and Running the Elastic Stack.

The code for this chapter can be found in the GitHub repository for the book:

https://github.com/PacktPublishing/Getting-Started-with-Elastic-Stack-8.0/tree/main/Chapter8

Navigate to Chapter8/trips-dataset in the code repository for this book and follow the instructions to load a dataset containing flight travel logs for a single passenger over a period of time:

1. The following files are provided in the folder:
   1. flights-template.json contains an Elasticsearch index template for the given dataset, detailing the schema/mappings for the fields in the dataset.
   2. load.sh is a helper script...

Getting up and running on Kibana

Collecting and ingesting data into your Elasticsearch cluster is only half the challenge when it comes to extracting insights and building useful outcomes from your datasets. Having access to fully featured and well-documented REST APIs on the Elasticsearch level is super useful, especially when your applications and systems programmatically consume responses from queries and aggregations, among other things. However, end users would much rather use an intuitive visual interface to build visualizations to understand trends in business data, diagnose bugs in their applications, and hunt for threats in their environment.

Kibana is the primary user interface when it comes to interacting with Elasticsearch clusters and, to some extent, components such as Logstash and Beats.

Given Kibana is primarily used to interact with data on Elasticsearch, an Elasticsearch cluster must be available for Kibana to run. The backing Elasticsearch cluster is used to...

Visualizing data with dashboards

Dashboards in Kibana are the primary tool to visualize datasets in order to understand what the data means. Users generally spend a significant chunk of their time on Kibana working with dashboards; well-designed dashboards can efficiently communicate important metrics, trends in activity, and any potential issues to look out for.

The Nginx dashboard shown in the following screenshot (available out of the box) visualizes source geo-locations, web server response codes over time, common error types, and top resources accessed on the web server. An engineer eyeballing this data can spot something out of the ordinary. If, for example, HTTP 5xx response codes suddenly start increasing for a given resource on the server, the engineer can quickly narrow down potential issues and proceed to fix them before end users are impacted:

Figure 8.5 – Nginx logs dashboard

Dashboards are designed to work interactively. Most visualizations...

Creating data-driven presentations with Canvas

Dashboards are a great way to visualize and consume data from Elasticsearch. Given their form factor, dashboards are interactive and can easily support analyst workflows in interrogating and pivoting data.

Dashboards, however, are not ideal when it comes to more granular control of how information is presented to a user. Canvas allows users to control the visual appearance of their data a lot more granularly, making it ideal for use in presenting key insights derived from data. Unlike normal presentations though, Canvas can be powered by live datasets on Elasticsearch in real time.

The following Canvas presentation presents some key insights from the trips dataset. A bunch of key stats, such as total trips, the number of countries, airlines, and total distance traveled, is rendered on the right side. The pie graph in the following Canvas presentation displays the proportion of business and economy class trips while the bubble chart...

Working with geospatial datasets using Maps

Elasticsearch comes with great support for geospatial data out of the box. Geo-point fields can hold a single geographic location (latitude/longitude pair) while Geo-shape fields support the encoding of arbitrary geoshapes (such as lines, squares, polygons, and so on). When searching for data on Elasticsearch, users can also leverage a range of geo queries, such as geo_distance (which finds docs containing a geo-point within a given distance from a specified geo_point) and geo_bounding_box (which finds docs with geo-points falling inside a specified geographical boundary). Kibana Maps is the visual interface for the geospatial capabilities on Elasticsearch.

Geospatial data is useful (and rather common) in several use cases. For example, logs containing public addresses will often contain (or can be enriched with) geo-location information for the corresponding host.

Analysts can use this context to understand whether connections to certain...

Responding to changes in data with alerting

So far in the chapter, we've looked at different ways in which users can interact with various types of data in real time. Analysts can easily explore and interrogate data and find events of interest and the consequences they may have on their use case.

Events of interest once discovered through analysis can happen multiple times in a system. Interactive analysis workflows involving a human do not necessarily scale in these cases, and there is a need to automate the detection of these events. This is where alerting plays an important role.

Kibana alerting is an integrated platform feature across all solutions in Kibana. Security analysts, for example, can use alerting to apply threat detection logic and the appropriate response workflows to mitigate potential issues. Engineering teams may use alerts to find precursors to a potential outage and alert the on-call site reliability engineer to take necessary action. We will explore...

Summary

In this chapter, we looked at how you can explore, analyze, and consume data on Elasticsearch using Kibana.

We started with learning how dashboards can be used to extract insights from large datasets. Then, we looked at how image-rich Canvas presentations, backed by live data can be a powerful visualization tool. Next, we looked at how Kibana Maps can help when working with geospatial datasets. We finished by exploring the use of Kibana alerting and actions to respond to changes in datasets.

The next chapter explores the management and continuous onboarding of data using Elastic Agent and Fleet.