Getting Data into Kibana

In this chapter, we'll cover different methods to push data into Kibana, such as from RDBMS, from different files, from system metrics, using CSV files, and from different applications. We'll start with different Beats, such as Packetbeat, Metricbeat, and Filebeat, to demonstrate the complete process, from configuring these Beats and then sending data from Beats to Elasticsearch or Logstash. Then we'll look at how to import data from CSV by configuring Logstash to take input and insert data into Elasticsearch. After CSV we'll fetch data from RDBMS using SQL queries through the JDBC plugin and insert it into Elasticsearch. For all these methods, we're inserting data into Elasticsearch. Once the data is inserted into Elasticsearch, we can configure Kibana to fetch the Elasticsearch data by creating an index pattern. In this way,...

Difference between Beats and Logstash

Beats are basically lightweight data shippers that are designed for a specific purpose, while Logstash is more generic and can be configured for multiple use cases. Beats have a smaller footprint, while Logstash has a larger footprint. We have different Beats for different purposes, such as Filebeat for handling files, Metricbeat for capturing system metrics, Packetbeat to capture network packet data, while Logstash has different plugins for input, filter, and output. We can read CSV data, RDBMS data, Beat data, or any other third-party application's data in Logstash, and after transformation can be sent to multiple sources. So basically, if we want to do any sort of data processing or want to capture any complex data, which isn't possible using Beats, we must use Logstash. If we just want to read log data, system metrics data, or...

Configuring Beats to get data

Here, we'll cover how to configure different Beats to get data, such as using Filebeat to capture file data, Metricbeat to capture system metrics data, and Metricbeat to capture network packet data. Using Beats in this way means we can start getting data, as these Beats are installed on a machine from where we want to capture the data, and once they're configured, they allow you to capture the data from the machine and send it to an Elasticsearch cluster directly or via Logstash. Beats are basically lightweight data shippers built for a specific purpose. There are different types of Beats, such as Packetbeat, Filebeat, Metricbeat, Auditbeat, Heartbeat, and Winlogbeat. The following diagram shows how these Beats are sending data to a central Elasticsearch Cluster:

In the preceding diagram, we have Filebeat, which is reading log data from...

Configuring Logstash to get data

In the previous section, we covered how to get data in Elasticsearch using different Beats. They're easy to install, configure, and then you can start receiving data from the server. Sometimes we need to do more than just configure a specific, single-purpose Beats that sits on the server and sends data to an Elasticsearch cluster and for the Logstash that's there. Logstash is a data pipeline we can use to configure input to take data from multiple types of data sources, such as files, databases, CSV, or Kafka, and after taking the input, we can configure the output to send data on different sources, such as files, databases, Kafka, or Elasticsearch. Another important feature of Logstash is filter, using which we can transform the input data before sending it to the output. Let's check out a Logstash configuration format:

input 
{
...

Configuring index patterns in Kibana

So far, we've covered how to push data into Elasticsearch from different sources, such as Beats and Logstash. Once the data is pushed into your Elasticsearch cluster, you can get that into Kibana by creating the index pattern. So the ultimate aim of Elastic Stack is to get data, analyze it, visualize it, and extract meaningful information – for that, we can fetch data from different sources, push them to a central Elasticsearch cluster, make them available to Kibana using index patterns, and analyze them.

Let's check how the index pattern can be created in Kibana to access Elasticsearch index data. Click on the Management link from the left menu to open the Management page. This will show you Elasticsearch and the Kibana block; under the Kibana block, click on the Index Pattern link to open the index pattern page. The following...

Summary

In this chapter, we looked at how to get data from different sources into Kibana, and then perform data analytics and visualization on it. We covered different ways to get data in Elasticsearch. We saw how Beats, a lightweight data shipper, can be installed on a server to send data. Under Beats, we covered Filebeat, which reads file data, such as Apache logs, system logs, and application logs, and can send them to Elasticsearch directly or using Logstash. Then we configured Metricbeat to read system metrics, such as CPU usage, memory usage, MySQL metrics, and then Packetbeat, using which we can read network packet data to get insights from it.

We covered how Logstash can be used to get data and apply filters before sending it to Elasticsearch. In the first section, we covered how to fetch CSV data using Logstash, where we passed a CSV file as input and specified the columns...