Chapter 6. Building Data Pipelines with Logstash

 

In the previous chapter, we understood the importance of Logstash in the log analysis process. We also covered its usage and its high-level architecture, and went through some commonly used plugins. One of the important processes of Logstash is converting unstructured log data into structured data, which helps in searching for relevant information easily and also assists in analysis. Apart from parsing the log data to make it structured, it would also be helpful if we could enrich the log data during this process so that we can gain further insight about our logs. Logstash comes in handy for enriching our log data, too. Also, we have seen in the previous chapter that Logstash can read from a wide range of inputs and that Logstash is a heavy process. Installing Logstash on the edge nodes of shipping logs might not always be feasible. Is there an alternative or lightweight agent that can be used to ship logs? Let's explore that in this chapter...

Parsing and enriching logs using Logstash

The analysis of structured data is easier and helps us find meaningful/deeper analysis, rather than trying to perform analysis on unstructured data. Most analysis tools depend on structured data. Kibana, which we will be making use of for analysis and visualization, can be used effectively if the data in Elasticsearch is right (the information in the log data is loaded into appropriate fields, and the data type of the fields are more appropriate than just having all the values of the log data in a single field). 

Log data is typically made up of two parts:

logdata = timestamp + data

timestamp is the time when the event occurred and data is the information about the event. data may contain just a single piece of information or it may contain many pieces of information. For example, if we take apache-access logs, the data piece will contain the response code, request URL, IP address, and so on. We would need to have a mechanism for extracting this information...

Introducing Beats

Beats are lightweight data shippers that are installed as agents on edge servers to ship operational data to Elasticsearch. Just like Elasticsearch, Logstash, Kibana, and Beats are open source products too. Depending on the use case, Beats can be configured to ship the data to Logstash for transforming the events prior to pushing the events to Elasticsearch. 

The Beats framework is made up of a library called libbeat which provides infrastructure to simplify the process of shipping the operation data to Elasticsearch. It offers the API that all Beats can use to ship data to an output (such as Elasticsearch, Logstash, Redis, Kafka, and so on), configure the input/output options, process the events, implement logging, and more. The Libbeat library is built using the Go programming language. Go was chosen to build Beats because it's easy to learn, very resource friendly, and as it's statically compiled it's easy to deploy.

Elastic.co has built and maintained several Beats such...

Filebeat

Filebeat is an open source, lightweight log shipping agent that is installed as an agent to ship logs from local files. It monitors the log directories or specific log files, tails the files, and forwards them to Elasticsearch, Logstash, Redis, or Kafka. It is salable and provides the ability to ship logs from multiple systems to a centralized system/server, from which the logs can be parsed and processed.

Downloading and installing Filebeat

Navigate to https://www.elastic.co/downloads/beats/filebeat and, depending on your operating system, download the .zip/.tar file. The installation of Filebeat is simple and straightforward:

Note

Beats version 6.0.x is compatible with Elasticsearch 5.6.x and 6.0.x, and Logstash 5.6.x and 6.0.x. The compatibility matrix can be found at https://www.elastic.co/support/matrix#matrix_compatibility. When you come across Elasticsearch and Logstash examples or usages with Beats in the chapter, make sure you have compatible versions of Elasticsearch and Logstash...

Summary

In this chapter, we have covered the powerful filter section of Logstash that can be used for parsing and enriching log events. We have also covered some of the commonly used filter plugins. We also covered the Beats framework and looked at an overview of various beats including Filebeat, Heartbeat, Packetbeat, and so on, and covered Filebeat in detail.

In the next chapter, we will be covering the various features of X-Pack, a commercial offering by Elastic.co which contains features such as the security to secure Elastic stack, monitoring, alerting, graphs, reporting, and many more.