Monitoring Microservices

In this chapter, we will learn how to use Prometheus and Grafana to collect, monitor, and alert about performance metrics. As we mentioned in Chapter 1, Introduction to Microservices, in a production environment it is crucial to be able to collect metrics for application performance and hardware resource usage. Monitoring these metrics is required to avoid long response times or outages for API requests and other processes.

To be able to monitor a system landscape of microservices in a cost-efficient and proactive way, we must also be able to define alarms that are triggered automatically if the metrics exceed the configured limits.

In this chapter, we will cover the following topics:

• Introduction to performance monitoring using Prometheus and Grafana
• Changes in source code to collect application metrics
• Building and deploying the microservices
• Monitoring microservices using Grafana dashboards
• Setting up alarms in...

Technical requirements

For instructions on how to install the tools used in this book and how to access the source code for this book, see:

• Chapter 21, Installation Instructions for macOS
• Chapter 22, Installation Instructions for Microsoft Windows with WSL 2 and Ubuntu

The code examples in this chapter all come from the source code in $BOOK_HOME/Chapter19.

If you want to view the changes applied to the source code in this chapter so that you can use Prometheus and Grafana to monitor and alert on performance metrics, you can compare it with the source code for Chapter 19, Centralized Logging with the EFK Stack. You can use your favorite diff tool and compare the two folders, $BOOK_HOME/Chapter19 and $BOOK_HOME/Chapter20.

Introduction to performance monitoring using Prometheus and Grafana

In this chapter, we will reuse the deployment of Prometheus and Grafana that we created in Chapter 18, Using a Service Mesh to Improve Observability and Management, in the Deploying Istio in a Kubernetes cluster section. Also in that chapter, we were briefly introduced to Prometheus, a popular open source database for collecting and storing time series data such as performance metrics. We learned about Grafana, an open source tool to visualize performance metrics. With the Grafana deployment comes a set of Istio-specific dashboards. Kiali can also render some performance-related graphs without the use of Grafana. In this chapter, we will get some hands-on experience with these tools.

The Istio configuration we deployed in Chapter 18 includes a configuration of Prometheus, which automatically collects metrics from Pods in Kubernetes. All we need to do is set up an endpoint in our microservice that produces metrics...

Changes in source code to collect application metrics

Spring Boot 2 introduced support for producing performance metrics in a Prometheus format using the Micrometer library (https://micrometer.io). There’s only one change we need to make to the source code of the microservices: we need to add a dependency on the Micrometer library, micrometer-registry-prometheus, in the Gradle build files, build.gradle. The dependency looks like this:

implementation 'io.micrometer:micrometer-registry-prometheus'

This will make the microservices produce Prometheus metrics on port 4004 using the /actuator/prometheus path.

In Chapter 18, we separated the management port, used by the actuator, from the port serving requests to APIs exposed by a microservice. See the Observing the service mesh section for a recap, if required.

To let Prometheus know about these endpoints, each microservice’s Pod is annotated with the following code:

annotations:...

Building and deploying the microservices

Building, deploying, and verifying the deployment using the test-em-all.bash test script is done in the same way it was done in Chapter 19, Centralized Logging with the EFK Stack, in the Building and deploying the microservices section. Run the following commands:

1. Build the Docker images from the source with the following commands:

   cd $BOOK_HOME/Chapter20
   eval $(minikube docker-env -u)
   ./gradlew build
   eval $(minikube docker-env)
   docker-compose build
   

The eval $(minikube docker-env -u) command ensures that the ./gradlew build command uses the host’s Docker engine and not the Docker engine in the Minikube instance. The build command uses Docker to run test containers.

2. Recreate the namespace, hands-on, and set it as the default namespace:

   kubectl delete namespace hands-on
   kubectl apply -f kubernetes/hands-on-namespace.yml
   kubectl config set-context $(kubectl config current...

Monitoring microservices using Grafana dashboards

As we already mentioned in the introduction, Kiali provides some very useful dashboards out of the box. In general, they are focused on application-level performance metrics such as requests per second, response times, and fault percentages to process requests. As we will see shortly, they are very useful on an application level. But if we want to understand the usage of the underlying hardware resources, we need more detailed metrics, for example, Java VM-related metrics.

Grafana has an active community that, among other things, shares reusable dashboards. We will try out a dashboard from the community that’s tailored to get a lot of valuable Java VM-related metrics from a Spring Boot application such as our microservices. Finally, we will see how we can build our own dashboards in Grafana. But let’s start by exploring the dashboards that come out of the box in Kiali and Grafana.

Before we do that, we need to...

Setting up alarms in Grafana

Being able to monitor the circuit breaker and retry metrics is of great value, but even more important is the capability to define automated alarms on these metrics. Automated alarms relieve us from monitoring the metrics manually.

Grafana comes with built-in support to define alarms and send notifications to a number of channels. In this section, we will define alerts on the circuit breaker and configure Grafana to send emails to the test mail server when alerts are raised. The local test mail server was installed earlier in the Installing a local mail server for tests section.

For other types of channels supported by the version of Grafana used in this chapter, see https://grafana.com/docs/grafana/v7.2/alerting/notifications/#list-of-supported-notifiers.

In the next section, we will define a mail-based notification channel that will be used by the alert in the section after this.

Setting up a mail-based notification channel...

Summary

In this chapter, we learned how to use Prometheus and Grafana to collect and monitor alerts on performance metrics.

We saw that, to collect performance metrics, we can use Prometheus in a Kubernetes environment. We then learned how Prometheus can automatically collect metrics from a Pod when a few Prometheus annotations are added to the Pod’s definition. To produce metrics in our microservices, we used Micrometer.

Then, we saw how we can monitor the collected metrics using dashboards in both Kiali and Grafana, which comes with the installation of Istio. We also experienced how to consume dashboards shared by the Grafana community, and learned how to develop our own dashboards, where we used metrics from Resilience4j to monitor the usage of its circuit breaker and retry mechanisms. Using the Grafana API, we can export created dashboards and import them into other Grafana instances.

Finally, we learned how to define alerts on metrics in Grafana and how to...

Questions

1. What changes did we need to make to the source code in the microservices to make them produce metrics that are consumed by Prometheus?
2. What is the management.metrics.tags.application config parameter used for?
3. If you want to analyze a support case regarding high CPU consumption, which of the dashboards in this chapter would you start with?
4. If you want to analyze a support case regarding slow API responses, which of the dashboards in this chapter would you start with?
5. What is the problem with counter-based metrics such as Resilience4j’s retry metrics, and what can be done so that we can monitor them in a useful way?
6. What is going on here?

Figure 20.23: What is going on here?

If you are reading this with screenshots rendered in grayscale, it might be hard to figure out what the metrics say. So, here’s some help:

1. The state transitions reported by the circuit breaker are, in order:
   1. half_open...