Centralized Logging with the EFK Stack

In this chapter, we will learn how to collect and store log records from microservice instances, as well as how to search and analyze log records. As we mentioned in Chapter 1, Introduction to Microservices, it is difficult to get an overview of what is going on in a system landscape of microservices when each microservice instance writes log records to its local filesystem. We need a component that can collect the log records from the microservice’s local filesystem and store them in a central database for analysis, search, and visualization. A popular open-source-based solution for this is based on the following tools:

• Elasticsearch, a distributed database with great capabilities for searching and analyzing large datasets
• Fluentd, a data collector that can be used to collect log records from various sources, filter and transform the collected information, and finally send it to various consumers, for example, Elasticsearch...

Technical requirements

For instructions on how to install the tools used in this book and how to access the source code for this book, see:

• Chapter 21, Installation Instructions for macOS
• Chapter 22, Installation Instructions for Microsoft Windows with WSL 2 and Ubuntu

The code examples in this chapter all come from the source code in $BOOK_HOME/Chapter19.

If you want to view the changes applied to the source code in this chapter, that is, see the changes we made so that we can use the EFK stack for centralized log analysis, you can compare it with the source code for Chapter 18, Using a Service Mesh to Improve Observability and Management. You can use your favorite diff tool and compare the two folders, $BOOK_HOME/Chapter18 and $BOOK_HOME/Chapter19.

Introducing Fluentd

In this section, we will learn the basics of how to configure Fluentd. Before we do that, let’s learn a bit about the background of Fluentd and how it works at a high level.

Overview of Fluentd

Historically, one of the most popular open source stacks for handling log records has been the ELK stack from Elastic (https://www.elastic.co), based on Elasticsearch, Logstash (used for log collection and transformation), and Kibana. Since Logstash runs on a Java VM, it requires a relatively large amount of memory. Over the years, a number of open source alternatives have been developed that require significantly less memory than Logstash, one of them being Fluentd (https://www.fluentd.org).

Fluentd is managed by the Cloud Native Computing Foundation (CNCF) (https://www.cncf.io), the same organization that manages the Kubernetes project. Therefore, Fluentd has become a natural choice as an open-source-based log collector that runs in Kubernetes. Together...

Deploying the EFK stack on Kubernetes

Deploying the EFK stack on Kubernetes will be done in the same way as we have deployed our own microservices: using Kubernetes manifest files for objects such as Deployments, Services, and ConfigMaps.

The deployment of the EFK stack is divided into three parts:

• Deploying Elasticsearch and Kibana
• Deploying Fluentd
• Setting up access to Elasticsearch and Kibana

But first, we need to build and deploy our own microservices.

Building and deploying our microservices

Building, deploying, and verifying the deployment using the test-em-all.bash test script is done in the same way as it was done in Chapter 18, Using a Service Mesh to Improve Observability and Management, in the Running commands to create the service mesh section. These instructions assume that the cert-manager and Istio are installed as instructed in Chapters 17 and 18.

Run the following commands to get started:

1. First, build the...

Trying out the EFK stack

The first thing we need to do before we can try out the EFK stack is to initialize Kibana so it knows what indices to use in Elasticsearch.

An index in Elasticsearch corresponds to a database in SQL concepts. The SQL concepts table, row, and column correspond to type, document, and property in Elasticsearch.

Once that is done, we will try out the following common tasks:

1. We will start by analyzing what types of log records Fluentd has collected and stored in Elasticsearch. Kibana has a very useful visualization capability that can be used for this.
2. Next, we will learn how to find all related log records created by the microservices while processing an external request. We will use the trace ID in the log records as a correlation ID to find related log records.
3. Finally, we will learn how to use Kibana to perform root cause analysis, finding the actual reason for an error.

Initializing Kibana

Before we start...

Summary

In this chapter, we learned about the importance of collecting log records from microservices in a system landscape into a common centralized database where analysis and searches of the stored log records can be performed. We used the EFK stack, Elasticsearch, Fluentd, and Kibana, to collect, process, store, analyze, and search for log records.

Fluentd was used to collect log records not only from our microservices but also from the various supporting containers in the Kubernetes cluster. Elasticsearch was used as a text search engine. Together with Kibana, we saw how easy it is to get an understanding of what types of log records we have collected.

We also learned how to use Kibana to perform important tasks such as finding related log records from cooperating microservices and how to perform root cause analysis, finding the real problem for an error message.

Being able to collect and analyze log records in this way is an important capability in a production environment...

Join our book community on Discord

https://packt.link/EarlyAccess

In this chapter, you will be introduced to the concept of a service mesh and see how its capabilities can be used to handle challenges in a system landscape of microservices, in areas including security, policy enforcement, resilience, and traffic management. A service mesh can also be used to provide observability, the capability to visualize how traffic flows between microservices.A service mesh overlaps partly with the capabilities of Spring Cloud and Kubernetes that we learned about earlier in this book. But most of the functionality in a service mesh complements Spring Cloud and Kubernetes, as we will see in this chapter.The following topics will be covered in this chapter:

• An introduction to the service mesh concept and Istio, a popular open source implementation
• Deploying Istio in Kubernetes
• Creating, observing, and securing a service mesh
• Ensuring that a service mesh is resilient
• Performing zero-downtime updates...