Chapter 5. System Monitoring

The previous two chapters focused on Elasticsearch monitoring tools, including Elasticsearch-head, Bigdesk, and Marvel. This chapter will introduce another monitoring tool, Kopf. We will also discuss Elasticsearch, Logstash, and Kibana (ELK), Nagios, and various GNU/Linux command line tools in terms of general purpose system monitoring.

This chapter covers these topics:

• Monitoring Elasticsearch with Kopf

• Configuring an Elasticsearch, Logstash, and Kibana (ELK) stack for system log file aggregation and analysis

• System-level monitoring of a cluster using Nagios

• GNU/Linux command line tools for system and process management

Working with Kopf

Kopf is a web-based cluster management tool like Elasticsearch-head, but has a more modern look and a few different features. With Kopf, users can check the state of nodes and indices, run REST queries, and perform basic management tasks.

Installing Kopf

Kopf works on Elasticsearch 0.90.x and up. Use the following table to determine which Kopf version is best suited to your cluster:

Elasticsearch Version	Kopf Branch
0.90.x	0.90
1.x	1.0
2.x	2.0

To install Kopf, follow these steps:

1. Install Kopf on at least one node in your cluster as an Elasticsearch plugin with the following command, replacing {branch} with the value from the branch column in the preceding table:

   $ sudo /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf/{branch}
   

   This example will install Kopf on elasticsearch-node-01. Since this node is running Elasticsearch 2.3.2, the command will look like this:

   $ sudo /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf/2.0
   

2. To...

Working with Logstash and Kibana

Logstash is a utility for aggregating and normalizing log files from disparate sources and storing them in an Elasticsearch cluster. Once logs are stored in Elasticsearch, we will use Kibana, the same tool Marvel's user interface is built on, to view and explore our aggregated logs.

ELK

The Elasticsearch community refers to the Elasticsearch, Logstash, and Kibana tool combination as the ELK stack. This section shows how to load NGINX server logs into ELK, but there are many other potential use cases for these technologies.

ELK can help us explore NGINX server logs by:

• Visualizing server traffic over time

• Plotting server visits by location on a map

• Searching logs by resource extension (HTML, JS, CSS, and so on), IP address, byte count, or user-agent strings

• Discovering web requests that result in internal server errors

• Finding attackers in a distributed denial of service attack

Other uses for ELK include:

• Logging all Elasticsearch queries in a web application for future...

Working with Nagios

Nagios is a system monitoring and alerting tool. This section will focus on configuring a simple Nagios installation that monitors the nodes in our Elasticsearch cluster, as well as the Elasticsearch process on those. If a node or process shuts down, Nagios will send us an alert.

It's a good idea to install Nagios on a host outside of the Elasticsearch clusters in order to avoid affecting the monitoring process due to other things going on in the system, such as high Elasticsearch load. Create a new host for Nagios and call it elasticsearc h-nagios-01.

Installing Nagios

In addition to the dedicated Nagios host, elasticsearch-nagios-01, install the Nagios Remote Plugin Executor (NRPE) server on all of the Elasticsearch cluster nodes in order to monitor the Elasticsearch process. Follow these steps:

1. Run the following command on each of the Elasticsearch nodes: elasticsearch-node-01, elasticsearch-node-02, elasticsearch-node-03, and elasticsea rch-marvel-01:

   sudo apt-get install...

Command line tools for system and process management

The command line is an invaluable tool for system monitoring. In this section, we'll go over a few basic GNU/Linux command line utilities for system and process management. Knowing these tools is essential for anyone managing an Elasticsearch cluster on GNU/Linux.

top

The top command lists processes with the highest CPU and memory. This tool is useful to determine whether a process other than Elasticsearch is hogging resources, or to check whether Elasticsearch is using an abnormal amount of CPU or memory.

The top command refreshes automatically, so you only have to run it once and watch.

When running the command, you should see the following result:

The top command

Tip

Press Shift+M while top is running to sort processes by those using the most memory instead of CPU.

tail

The tail -f command is useful for viewing log files in real time. Use it to view Elasticsearch log files as follows:

tail -f /var/log/elasticsearch/*

"tailing" Elasticsearch log...

Summary

This chapter examined the Elasticsearch monitoring tool Kopf, the Elasticsearch, Logstash, and Kibana (ELK) log aggregation stack, the system monitoring tool Nagios, and various GNU/Linux command line utilities.

Some takeaways are:

• Kopf is an Elasticsearch monitoring tool similar to Elasticsearch-head, but provides a few different metrics.

• The Elasticsearch, Logstash, and Kibana (ELK) stack is a tool for searching, analyzing, enriching, and visualizing log files.

• Consider using a tool such as Nagios to monitor an Elasticsearch cluster. Nagios can be configured to send out email notifications when a process goes down or if the node itself goes down.

• Using a few GNU/Linux command tools, we can gather many of the same metrics provided by the various Elasticsearch monitoring tools.

The next chapter will discuss troubleshooting Elasticsearch performance and reliability issues. The monitoring tools discussed in this chapter will be useful when tackling the real-world problems outlined in upcoming...

The rest of the chapter is locked

Working with Logstash and Kibana

Logstash is a utility for aggregating and normalizing log files from disparate sources and storing them in an Elasticsearch cluster. Once logs are stored in Elasticsearch, we will use Kibana, the same tool Marvel's user interface is built on, to view and explore our aggregated logs.

ELK

The Elasticsearch community refers to the Elasticsearch, Logstash, and Kibana tool combination as the ELK stack. This section shows how to load NGINX server logs into ELK, but there are many other potential use cases for these technologies.

ELK can help us explore NGINX server logs by:

• Visualizing server traffic over time

• Plotting server visits by location on a map

• Searching logs by resource extension (HTML, JS, CSS, and so on), IP address, byte count, or user-agent strings

• Discovering web requests that result in internal server errors

• Finding attackers in a distributed denial of service attack

Other uses for ELK include:

• Logging all Elasticsearch queries in a web application for future...

Working with Nagios

Nagios is a system monitoring and alerting tool. This section will focus on configuring a simple Nagios installation that monitors the nodes in our Elasticsearch cluster, as well as the Elasticsearch process on those. If a node or process shuts down, Nagios will send us an alert.

It's a good idea to install Nagios on a host outside of the Elasticsearch clusters in order to avoid affecting the monitoring process due to other things going on in the system, such as high Elasticsearch load. Create a new host for Nagios and call it elasticsearc h-nagios-01.

Installing Nagios

In addition to the dedicated Nagios host, elasticsearch-nagios-01, install the Nagios Remote Plugin Executor (NRPE) server on all of the Elasticsearch cluster nodes in order to monitor the Elasticsearch process. Follow these steps:

1. Run the following command on each of the Elasticsearch nodes: elasticsearch-node-01, elasticsearch-node-02, elasticsearch-node-03, and elasticsea rch-marvel-01:

   sudo apt-get install...

Command line tools for system and process management

The command line is an invaluable tool for system monitoring. In this section, we'll go over a few basic GNU/Linux command line utilities for system and process management. Knowing these tools is essential for anyone managing an Elasticsearch cluster on GNU/Linux.

top

The top command lists processes with the highest CPU and memory. This tool is useful to determine whether a process other than Elasticsearch is hogging resources, or to check whether Elasticsearch is using an abnormal amount of CPU or memory.

The top command refreshes automatically, so you only have to run it once and watch.

When running the command, you should see the following result:

The top command

Tip

Press Shift+M while top is running to sort processes by those using the most memory instead of CPU.

tail

The tail -f command is useful for viewing log files in real time. Use it to view Elasticsearch log files as follows:

tail -f /var/log/elasticsearch/*

"tailing" Elasticsearch log...

Summary

This chapter examined the Elasticsearch monitoring tool Kopf, the Elasticsearch, Logstash, and Kibana (ELK) log aggregation stack, the system monitoring tool Nagios, and various GNU/Linux command line utilities.

Some takeaways are:

• Kopf is an Elasticsearch monitoring tool similar to Elasticsearch-head, but provides a few different metrics.

• The Elasticsearch, Logstash, and Kibana (ELK) stack is a tool for searching, analyzing, enriching, and visualizing log files.

• Consider using a tool such as Nagios to monitor an Elasticsearch cluster. Nagios can be configured to send out email notifications when a process goes down or if the node itself goes down.

• Using a few GNU/Linux command tools, we can gather many of the same metrics provided by the various Elasticsearch monitoring tools.

The next chapter will discuss troubleshooting Elasticsearch performance and reliability issues. The monitoring tools discussed in this chapter will be useful when tackling the real-world problems outlined in upcoming...