Chapter 5. Using Beats

In the previous chapters, we explored the former ELK Stack components, along with the basics of each component and how they function together. Now, as the Stack expands and grows rapidly, let's familiarize ourselves with the newest component in the Elastic Stack.

In this chapter, we will understand the basic foundation of Beats and cover various other sections focusing entirely on Beats. We will cover from the basics to the advanced level, showcasing the ability of different types of Beats and how Beats fits into the Elastic Stack.

In this chapter, we will cover the following sections:

• Introduction to Beats

• How Beats differs from Logstash

• How Beats fits into Elastic Stack

• An overview of the different types of Beats

• Exploring Elastic Team Beats

• Exploring Community Beats

• Beats in action with Elastic Stack

Introduction to Beats

Before learning about Beats, let's understand how Beats first came into existence. Beats was not started as a project within Elastic, the company behind the ELK Stack. It all started with the Packetbeat project, wherein they wanted access to statistics between communications among various servers, providing them information.

They wanted to develop a solution which would store data in Elasticsearch, but would not consume high memory or resources. They faced the problem of using a Logstash agent on each server from which information was required, which used JVM and consumed a lot of memory and resources. Packetbeat became popular due to its ability to provide lightweight shippers, which ship the data to Elasticsearch without consuming much memory and CPU resources. Following its popularity, Elastic acquired Packetbeat and created the Beats Platform. The Beats Platform contains a number of Beats which have been developed by Elastic; this also contains Community Beats, which...

How Beats differ from Logstash

From what we have understood about Beats, it looks fairly similar to Logstash, wherein we have input plugins that are used to parse the data. So, how does Beats differ? Beats are data shippers shipping data from a variety of inputs such as files, data streams, or logs whereas Logstash is a data parser. Though Logstash can ship data, it's not its primary usage. In a nutshell, Beats and Logstash are similar in functionality, but there are glaring differences between them both in terms of how they are developed and the underlying technology used.

Let's look at the differences between them:

• Logstash consumes a lot of memory and requires a higher amount of resources, whereas Beats requires fewer resources and consumes low memory.

• Logstash is heavy to install on all the systems from which you want to collect the logs, whereas Beats are lightweight data shippers that will ship your data from multiple systems.

• If you are collecting logs from multiple systems, then you...

How Beats fits into Elastic Stack

Having used the ELK Stack, which has been in existence for some time, we ask: is there space for another component in the Stack?

This is one of the biggest questions we think you'll have: where and how will Beats fit into the Elastic Stack? As we have already covered what Beats are, we have your answer covered, and this section that will provide you the answer to your question.

Beats has been added among the core components of the Stack due to the endless opportunities it creates when you use it. In the ELK Stack, you are bounded by the input plugins provided, via which only you can read the data. If you want to index operational data within Elasticsearch, such as transaction level information between multiple systems, Docker container statistics, Tomcat JMX metrics, or system-wide process level statistics, you would need to write a Logstash input plugin, which would then be used for such scenarios. For scenarios where you have multiple systems and you want...

An overview of the different types of Beats

In this section, we will look at some of the beats that have been developed and are in use by enterprises for development and production purposes. Beats are being developed and enhanced by Elastic as well as by the community.

The following are some of the beats developed by Elastic:

• Packetbeat

• Metricbeat

• Filebeat

• Winlogbeat

• Libbeat

The following are some of the beats developed by the open source community:

• Apachebeat

• Dockbeat

• Elasticbeat

• Execbeat

• Httpbeat

• Lmsensorsbeat

• MySQLbeat

• Twitterbeat and more

Note

Elastic does not provide any support for the open source community developed Beats.

Beats by Elastic Team

As mentioned earlier, we will take a look at the different types of beats provided by Elastic Team in order to understand what each type of beat means and what functionalities they offer.

Packetbeat

Packetbeat is the main beat that started as a project outside Elastic, and the company that Elastic acquired for creating their Beats Platform has become an integral...

Exploring Elastic Team Beats

As we already covered the different types of beats provided by the Elastic Team, in this section, we will explore those beats and understand their features, learn how to set them up, and how to configure the beats with the various configuration options as provided. This section will provide you with all the necessary information you will need for understanding beats in detail.

Understanding Filebeat

As discussed, Filebeat is an open source log shipping agent that has been inspired by the Logstash-forwarder project, and it's based on the Logstash-forwarder source code. Filebeat is used for monitoring logs (files or directories) and forwards those logs to either Logstash, for further processing, or to Elasticsearch, for data indexing.

Let's understand how Filebeat works with the help of its architecture, as displayed below:

Link: https://www.elastic.co/guide/en/beats/filebeat/5.1/filebeat-overview.html

In the preceding image, there are a lot of new terms to understand...

Exploring Community Beats

As Elastic Stack components are parts of an open source initiative, Beats have been embraced by the community, which is actively contributing to the creation of different types of Beats. There are many new beats being developed by the community that follow the Beats platform. Let us have a look at a community beat, explore it, set it up, and understand the configuration options provided for the beat.

Understanding Elasticbeat

This is a community-developed beat that provides simple yet useful functionalities. This beat provides Elasticsearch cluster related information. It provides statistics of elasticsearch clusters, elasticsearch nodes, and the health of elasticsearch nodes using the API. It requests various types of statistics from the elasticsearch cluster, which are exposed via the elasticsearch API.

Let's set up Elasticbeat in order to use it.

Installation of Elasticbeat

In this section, Elasticbeat will be installed, and this section covers installation on Ubuntu...

Beats in action with Elastic Stack

After learning about the various kinds of beats, their functionality, and the information provided, we will learn about how to get started with Beats in order to use it along with Elastic Stack. We will cover the following use cases, showcasing an end-to-end flow for the following cases:

• Using Metricbeat to directly send data to Logstash, which will process and store logs in Elasticsearch so that they can then be visualized in Kibana

• Using Elasticbeat to index data in Elasticsearch and visualize it using Kibana

In this section, you will understand how to get started with simple processing and how to connect multiple components with ease. The objective of this section is not to show you the powerful processing capabilities of Logstash, or how Beats will optimize performance across various systems; it is just to show you how we can connect multiple components so that data flows from each component, and we can corelate the input from Beats to visualize the output...

Summary

In the chapter, we learned about Beats which is the newest addition to the Elastic Stack. We covered the basic premises for why Beats was developed or came into existence along with differentiating Beats with Logstash. We explored the role of Beats in Elastic Stack and the importance of Beats for extending former ELK Stack. Later, we explored the different kind of Beats available with an explanation of the Beats, their functionalities, roles, and the configuration options provided. Also how to install and configure the Beats were detailed in this chapter. At the end of the chapter, we covered the example of how Beats can be used in Elastic Stack.

In the next chapter, we will cover a real-time production-like environment wherein we will be solving a particular problem statement. We will explore how Elastic Stack comes to the rescue of the problem statement and will showcase the powerful capabilities of Elastic Stack.