Mastering Kibana 6.x

Elasticsearch is a full text search engine that can be used as a NoSQL database and as an analytics engine. It is easy to scale, schemaless, and near real time, and provides a restful interface for different operations. It isschemaless,and it uses inverted indexes for data storage. There are different language clients available for Elasticsearch, as follows:

The basic components of Elasticsearch are as follows:

Logstash is basically used for data pipelining, through which we can take input from different sources and output to different data sources. Using Logstash, we can clean the data through filter options and mutate the input data before sending it to the output source. Logstash has different adapters to handle different applications, such as for MySQL or any other relational database connection. We have a JDBC input plugin through which we can connect to MySQL server, run queries, and take the table data as the input in Logstash. For Elasticsearch, there is a connector in Logstash that gives us the option to seamlessly transfer data from Logstash to Elasticsearch.

To run Logstash, we need to install Logstash and edit the configuration file logstash.conf, which consists of an input, output, and filter sections. We need to tell Logstash where it should get the input from through the input block, what it should do with the input through the filter block, and where it should send the output through the output block. In the following example, I am reading an Apache Access Log and sending the output to Elasticsearch:

input {
     file {
         path => "/var/log/apache2/access.log"
         }
 }

 filter {
     grok {
         match => { message => "%{COMBINEDAPACHELOG}" }
         }
 }

output {
   elasticsearch {
        hosts => "http://127.0.0.1:9200"
        index => "logs_apache"
        document_type => "logs"
        }
}

The input block is showing a file key that is set to /var/log/apache2/access.log. This means that we are getting the file input and path of the file, /var/log/apache2/access.log, which is Apache's log file. The filter block is showing the grok filter, which converts unstructured data into structured data by parsing it.

There are different patterns that we can apply for the Logstash filter. Here, we are parsing the Apache logs, but we can filter different things, such as email, IP addresses, and dates.

Kibana is a dashboarding open source software from ELK Stack, and it is a very good tool for creating different visualizations, charts, maps, and histograms, and by integrating different visualizations together, we can create dashboards. It is part of ELK Stack; hence it is quite easy to read the Elasticsearch data. This does not require any programming skills. Kibana has a beautiful UI for creating different types of visualizations, including charts, histograms, and dashboards.

It provides us with different inbuilt dashboards with multiple visualizations when we use Beats, as it automatically creates multiple visualizations that we can customize to create a useful dashboard, such as for CPU usage and memory usage.

Beats are basically data shippers that are grouped to do single-purpose jobs. For example, Metricbeat is used to collect metrics for memory usage, CPU usage, and disk space, whereas Filebeat is used to send file data such as logs. They can be installed as agents on different servers to send data from different sources to a central Logstash or Elasticsearch cluster. They are written in Go; they work on a cross-platform environment; and they are lightweight in design. Before Beats, it was very difficult to get data from different machines as there was no single-purpose data shipper, and we had to do some tweaking to get the desired data from servers.

For example, if I am running a web application on the Apache web server and want to run it smoothly, then there are two things that need to be monitored—first, all of the errors from the application, and second, the server's performance, such as memory usage, CPU usage, and disk space. So, in order to collect this information, we need to install the following two Beats on our machine:

To visualize this data, we can use Kibana to create meaningful dashboards through which we can get complete control of our data.

Elasticsearch 6.0 requires that we have Java 8 at the least. Before you proceed with the installation of Elasticsearch, please ensure which version of Java is present in your system by executing the following command:

java -version
echo $JAVA_HOME

After the setup is complete, we can go ahead and run Elasticsearch. You can find the binaries at www.elastic.co/downloads.

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.3.tar.gz

tar -xvf elasticsearch-6.1.3.tar.gz

cd elasticsearch-6.1.3/bin

./elasticsearch

brew install elasticsearch

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

sudo apt-get update && sudo apt-get install elasticsearch

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

[elasticsearch-6.x]
 name=Elasticsearch repository for 6.x packages
 baseurl=https://artifacts.elastic.co/packages/6.x/yum
 gpgcheck=1
 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
 enabled=1
 autorefresh=1
 type=rpm-md

sudo yum install elasticsearch

sudo dnf install elasticsearch

sudo zypper install elasticsearch

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

Logstash requires at least Java 8. Before you go ahead with the installation of Logstash, please check the version of Java in your system by running the following command:

java -version
echo $JAVA_HOME

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

sudo apt-get update && sudo apt-get install logstash

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

[logstash-6.x]
 name=Elastic repository for 6.x packages
 baseurl=https://artifacts.elastic.co/packages/6.x/yum
 gpgcheck=1
 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
 enabled=1
 autorefresh=1
 type=rpm-md

sudo yum install logstash

Starting with version 6.0.0, Kibana only supports 64-bit operating systems.

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.1.3-linux-x86_64.tar.gz

sha1sum kibana-6.1.3-linux-x86_64.tar.gz
tar -xzf kibana-6.1.3-linux-x86_64.tar.gz

cd kibana-6.1.3-linux-x86_64/

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

sudo apt-get update && sudo apt-get install kibana

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

[kibana-6.x]
 name=Kibana repository for 6.x packages
 baseurl=https://artifacts.elastic.co/packages/6.x/yum
 gpgcheck=1
 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
 enabled=1
 autorefresh=1
 type=rpm-md

sudo yum install kibana

sudo dnf install kibana

sudo zypper install kibana

CD c:\kibana-6.1.3-windows-x86_64

.\bin\kibana

After installing and configuring the ELK Stack, you need to install and configure your Beats.

Each Beat is a separately installable product. To get up and running quickly with a Beat, see the getting started information for your Beat:

sudo apt-get install libpcap0.8
curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-6.2.1-amd64.deb
 sudo dpkg -i packetbeat-6.2.1-amd64.deb

sudo yum install libpcap
 curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-6.2.1-x86_64.rpm
 sudo rpm -vi packetbeat-6.2.1-x86_64.rpm

curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-6.2.1-darwin-x86_64.tar.gz
 tar xzvf packetbeat-6.2.1-darwin-x86_64.tar.gz

PS > cd 'C:\Program Files\Packetbeat'
PS C:\Program Files\Packetbeat> .\install-service-packetbeat.ps1

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.2.1-amd64.deb
 sudo dpkg -i metricbeat-6.2.1-amd64.deb

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.2.1-x86_64.rpm
 sudo rpm -vi metricbeat-6.2.1-x86_64.rpm

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.2.1-darwin-x86_64.tar.gz
 tar xzvf metricbeat-6.2.1-darwin-x86_64.tar.gz

PS > cd 'C:\Program Files\Metricbeat'
PS C:\Program Files\Metricbeat> .\install-service-metricbeat.ps1

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.1-amd64.deb
 sudo dpkg -i filebeat-6.2.1-amd64.deb

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.1-x86_64.rpm
 sudo rpm -vi filebeat-6.2.1-x86_64.rpm

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.1-darwin-x86_64.tar.gz
 tar xzvf filebeat-6.2.1-darwin-x86_64.tar.gz

PS > cd 'C:\Program Files\Filebeat'
PS C:\Program Files\Filebeat> .\install-service-filebeat.ps1

PS C:\Users\Administrator> cd 'C:\Program Files\Winlogbeat'
PS C:\Program Files\Winlogbeat> .\install-service-winlogbeat.ps1

Do you want to run
 C:\Program Files\Winlogbeat\install-service-winlogbeat.ps1?
 [D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"): R

 Status Name DisplayName
 ------ ---- -----------
 Stopped winlogbeat winlogbeat

curl -L -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-6.2.1-amd64.deb
 sudo dpkg -i heartbeat-6.2.1-amd64.deb

curl -L -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-6.2.1-x86_64.rpm
 sudo rpm -vi heartbeat-6.2.1-x86_64.rpm

curl -L -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-6.2.1-darwin-x86_64.tar.gz
 tar xzvf heartbeat-6.2.1-darwin-x86_64.tar.gz

PS > cd 'C:\Program Files\Heartbeat'
PS C:\Program Files\Heartbeat> .\install-service-heartbeat.ps1

In any large organization, there will be different servers with different sets of applications. So, in this case, we need to have different teams for different applications whose task is to explore the log files for debugging any issue. However, this is not an easy task, as the format of logs is never user friendly. Here, I am talking about a single application, but what will happen if we ask the team to monitor all different applications that are built using different technologies and their log format is very different from other applications? The answer is very simple: the team has to dig through all the logs from the different servers and then they will spend days and nights to find the issue. ELK Stack is very useful for these situations, and we can solve this problem easily. First of all, we need to set up a central Elasticsearch cluster for collecting all different logs. Now, we need to configure Logstash as per the application log so that we can transform different log formats that we are getting from different application servers. Logstash will output this data into Elasticsearch for storage so that we can explore, search, and update the data. Finally, Kibana can be used to display graphical dashboards on top of Elasticsearch. Using this setup, anyone can get complete control of all logs coming from different sources. We can use Kibana to alert us to any issues in the log file so that the user can get the issue without doing any data drill downs. Many organizations are using ELK for their log management as this is an open source software that can be built easily to monitor different type of logs on a single screen. Not only can we monitor all of our logs in a single screen, but we can also get alerts if something went wrong in the logs.

Security monitoring and alerting is a very important use case of ELK Stack as application security is a vital part, and it costs if there are any security breaches in the application since security breaches are becoming more common, and most importantly, more targeted. Although enterprises are regularly trying to improve their security measures, hackers are successful in penetrating the security layers. Therefore, it is very much required for any enterprise to detect the presence of security attacks on their server, and not only detect but also alert them so that they can take immediate actions to mitigate their losses. Using ELK Stack, we can monitor various things, such as unusual server requests and any suspicious traffic. We can gather security-related log information that can be monitored by security teams to check any alerts to the system.

This way, security teams can prevent the enterprise from attackers who have gone unnoticed for a long time. ELK Stack provides a way through which we can gain an insight and make the attacker's life more difficult. These logs can also be very useful for after-attack analysis; for example, for finding out the time of the attack and the method of attack used. We can understand the activities the attacker performed to attack, and this information can provide us with a way to strengthen that loophole easily. In this way, ELK Stack is useful for both before attack prevention and after attack healing and prevention.

In ELK Stack, we have different tools to grab data from remote servers. In traditional Relational Database Management System (RDBMS), it is quite difficult to save these types of data because they are not structured, so either we have to manually clean the data or leave some part of it in order to save it in the table schema. In the case of Elasticsearch, the schemaless behavior gives us the leverage to push any data from any source. It not only holds that data but also provides us with a feature to search and play with it. An example of web scraping using ELK Stack is a Twitter to Elasticsearch connector, which allows us to set up hashtags from Twitter and grab all the tweets that used those hashtags. After grabbing those hashtags, we can search, visualize, and analyze them in Kibana.

Many of the top e-commerce websites, such as eBay's, are using Elasticsearch for their product search pages. The main reason behind this is the ability of Elasticsearch in full-text searching, building filters, facets, aggregations, fast response time, and the ease it provides in collecting analytic information. Users can easily drill down to get the product set, from where they can easily select the product they want. This is just one side of the picture, through which we are improving the user's experience. On the other side, we can use the same data and by using Kibana, we can monitor the trends, analyze the data, and much more. There is a big competition going on among e-commerce companies to attract more and more customers. Being able to understand the shopping behavior of their customers is a very important feature, as it leverages e-commerce companies to target users with products that they had liked or will like. This is business intelligence, and using ELK Stack, they can achieve it.

ELK Stack's core competency is its full text search feature. It is powerful and flexible, and it provides various features such as fuzzy search, conditional searching, and natural language searching. So, as per our requirements, we can decide which type of searching is required. We can use ELK Stack's full text search capabilities for product searching, autocomplete features, searching text in emails, and so on.

Kibana is an easy-to-use visualization tool that provides us with a rich feature set to create beautiful charts (such as pie charts, bar charts, and stack charts), histograms, geo maps, word tags, data tables, and so on. Visualizing data is always beneficial for any organization as it helps top management to make decisions with ease. We can also easily track any unusual trends and find any outliers in data without digging into the data. We can create dashboards for any existing web-based application as well by simply pushing the application data into Elasticsearch and then use Kibana to create beautiful dashboards. This way, we can plug in an additional dimension into the application and start monitoring it without putting any additional load on the application.