Chapter 11. Monitoring Server Infrastructure

In the previous chapter, we covered how to effectively run Elastic Stack in a production environment, and the best practices to follow when running Elastic Stack in production.

In this chapter, we will be covering how to use the Beats platform for monitoring server infrastructure. We will learn in detail about Metricbeat, a Beat which helps IT administrators and application support teams in monitoring their applications and server infrastructure, and in responding in a timely manner in case of infrastructure outage. 

In this chapter, we will cover:

• Metricbeat, a Beat used for collecting system and application metrics
• Installation and configuration of Metricbeat
• Deployment architectures 

Metricbeat

Metricbeat is a lightweight shipper that periodically collects metrics from the operating system and from services running on the server. It helps one to monitor servers by collecting metrics from the system and services such as Apache, MongoDB, Redis, and so on, running on the server. Metricbeat can push the collected metrics directly into Elasticsearch or send them to Logstash, Redis, or Kafka. To monitor services, Metricbeat can be installed on the edge server where services are running, but it also provides the ability to collect metrics from remote servers, as well. However, it's recommended to have it installed on the edge servers where the services are running. 

Downloading and installing Metricbeat

Navigate to https://www.elastic.co/downloads/beats/metricbeat and, depending on your operating system, download the ZIP/TAR file as shown in the following screenshot. The installation of Metricbeat is simple and straightforward:

Note

Beats version 6.0.x is compatible with Elasticsearch...

Configuring Metricbeat

The configurations related to Metricbeat are stored in a configuration file named metricbeat.yml, and it uses YAML syntax. 

The metricbeat.yml file contains the following:

• Module configuration
• General settings
• Output configuration 
• Processor configuration 
• Path configuration
• Dashboard configuration 
• Logging configuration

Let's explore some of these sections.

Note

The location of the metricbeat.yml file will be present in the installation directory if .zip or .tar files are used for installation. If .dep or .rpm is used for installation, then it will be present in the /etc/metricbeat location.

Module configuration

Metricbeat comes bundled with various modules to collect metrics from the system and applications such as Apache, MongoDB, Redis, MySQL, and so on.

Metricbeat provides two ways of enabling modules and metricsets:

• Enabling module configs in the modules.d directory
• Enabling module configs in the metricbeat.yml file

Enabling module configs in the modules.d directory

The modules...

Capturing system metrics

In order to monitor and capture metrics related to servers, Metricbeat provides the system module. The system module provides the following metricsets to capture server metrics:

• core: This metricset provides usage statistics for each CPU core.
• cpu: This metricset provides CPU statistics.
• diskio: This metricset provides disk IO metrics collected from the operating system. One event is created for each disk mounted on the system.
• filesystem: This metricset provides file system statistics. For each file system, one event is created.
• process: This metricset provides process statistics. One event is created for each process.
• process_summary: This metricset collects high-level statistics about the running processes.
• fsstat: This metricset provides overall file system statistics.
• load: This metricset provides load statistics.
• memory: This metricset provides memory statistics.
• network: This metricset provides network IO metrics collected from the operating system. One event is created...

 Deploymezs architecture

The following diagram depicts commonly used Elastic Stack deployment architecture:

The diagram depicts three possible architectures:

• Ship the operation metrics directly to Elasticsearch: As seen in the preceding diagram, one will install various types of Beats such as Metricbeat, Filebeat, Packetbeat, and so on, on the edge servers from which they would like to ship the operation metrics/logs. If no further processing of events is required, then the generated events can be shipped directly to the Elasticsearch cluster. Once the data is present in Elasticsearch, it can then be visualized/analyzed using Kibana. In this architecture, the flow of events would be Beats → Elasticsearch → Kibana. 
• Ship the operation metrics to Logstash: The operation metrics/logs captured by the Beats and installed on edge servers is sent to Logstash for further processing such as, for instance, parsing the logs or enriching log events. Then the parsed/enriched events are pushed to Elasticsearch...

Summary

In this chapter, we have covered in detail another Beat library called Metricbeat. We covered how to install and configure Metricbeat so that it can send operational metrics to Elasticsearch. We also covered the various deployment architectures for building real-time monitoring solutions using Elastic Stack, in order to monitor servers and applications. This helps IT administrators and application support folks gain insight into the behavior of the applications and servers and allows them to respond in a timely manner in case of infrastructure outage.