Using Kibana, Logstash, and Beats

In the last chapter, we introduced Python programming with Elasticsearch, and presented the low-level client and high-level library. We wrote a number of unit test programs and tried to implement Bollinger Bands with the high-level library. However, due to the fact that the moving_fn aggregation is not yet supported, we used a constant value for the moving standard deviation instead. If you cannot find the support function from the high-level client, you need to switch back to the low-level client. In this chapter, we will give an overview of the Elastic Stack's components, including Kibana, Logstash, and Beats. The Elastic Stack is a rich ecosystem. Knowing one piece of it is good. Two pieces are better, and three pieces are excellent. If we understand most of the components, our role could be as a fullstack engineer, and even an architect...

Overview of the Elastic Stack

According to the documentation of the latest version of the Elastic Stack (available at https://www.elastic.co/guide/en/elastic-stack/7.0/installing-elastic-stack.html), there are six components: Elasticsearch, Kibana, Logstash, Beats, APM Server, and Elasticsearch-Hadoop. If you use more than one component, you must use the same version across the stack. Let's give a brief description of each component as follows:

• Kibana: In version 7.0 of the Elastic Stack, Kibana is a UI framework for working with Elasticsearch. It now supports a dark theme across the entire UI, and a responsive dashboard to take a step forward to support mobile devices.
• Logstash: This is an open source data collection engine that provides a real-time data processing pipeline from multiple sources and then sends the collected logs to a defined target, such as an Elasticsearch...

Running Elasticsearch in a Docker container

On our GitHub site (https://github.com/PacktPublishing/Mastering-Elasticsearch-7.0/tree/master/Chapter13/docker_run), you can download a set of bash script files to run the application including docker_create_network, docker_run_elasticsearch, docker_run_kibana, docker_run_logstash, docker_run_filebeat, and a folder named pipeline, which contains the logstash.conf Logstash configuration file. Use the bash script file to launch an Elasticsearch server in development mode with the Elasticsearch Docker image by running the following command. However, previously you run the following command, shut down the original Elasticsearch server installed before and go to the docker_run folder:

$cd docker_run
$./docker_run_elasticsearch

You can check the status of the Elasticsearch Docker container to see whether it is running by using the list command...

Running Kibana in a Docker container

You can quickly launch Kibana with the Kibana Docker image. You can search, view, and analyze the Elasticsearch index in the forms of charts, tables, and maps. Run the following command to bring up Kibana:

$./docker_run_kibana

Direct your web browser to http://localhost:5601 to communicate with Kibana. The home page of Kibana is as follows:

Let's try to test Kibana with the sample data provided by the official website by following these steps:

1. Press Add sample data hyperlink and you will see the page shown in the following screenshot:

2. Now, l take the Sample flight data. Press the Add data button and you will see the page shown in the following screenshot:

3. Amazing! What a wonderful dashboard to show the flight data. Move your mouse to the button in the left toolbar, highlighted by the a box, and press it. This will take you to the...

Running Logstash in a Docker container

Logstash receives data from multiple sources, performs data processing, and then sends the log information to the stash, which can mean a store. There are two types of configurations with which to configure Logstash for Docker: pipeline configuration and the settings configuration. We will use pipeline configuration for our demonstration. When the Logstash Docker container runs with pipeline configuration, it will check the path for the logstash.conf file. In our case, the file path is docker_run/pipeline/logstash.conf, as specified in the docker_run_logstash script file. The structure of a Logstash configuration file basically includes three parts: input, filter, and output. You specify the source of the data in the input section, and the destination in the output section. You can manipulate, measure, and create events in the filter section...

Running Beats in a Docker container

Beats has a large, supportive, open source community. Besides the official GitHub site (https://github.com/elastic/beats), there are about 80 other GitHub repositories (see https://www.elastic.co/guide/en/beats/libbeat/7.0/community-beats.html) focused on supporting different types of data, such as Amazon products, Apache HTTPD, Spring Boot, and Kafka. Undoubtedly, Beats are a hot topic. All the Beats are built on top of the libbeat framework, which contains the common packages written in the Go language. The officially supported Beats and their target data are listed in this table:

Beat name	Type of data	Description
Auditbeat	Audit data	Install it on the servers where you want to detect changes to the attributes of files or the system's events and have them sent to an output destination.
Filebeat	Log files	A lightweight...

Summary

Bravo! We have tried the major components of the Elastic Stack. You should now understand the basic concepts of the powerful Elastic Stack. We ran an example on Kibana to visualize some sample flight data from Elasticsearch. We also learned how to use Logstash to collect and parse log data from the system log file. We extended the use of Logstash as a central log-processing center by using Filebeat. We also played with the popular deployment technique of running the applications using the officially supported Elastic Stack Docker images.

In the next chapter, we will introduce Elasticsearch SQL. Yes, Elasticsearch also speaks SQL. You will learn the SQL semantics supported in Elasticsearch. You will also perform a SQL REST API with SQL statements. Also, you will work with JDBC (Java Database Connectivity), the software industry standard for databases.

...