Chapter 7. Monitoring

When something goes wrong in a system, the concerned stakeholders will want to know what has happened, why it has happened and any hint or clue for fixing it, and how to prevent the same problem from occurring again in the future. This is one of the primary uses of monitoring. However, monitoring spans well beyond this primary usage.

In .NET monoliths, there are multiple monitoring solutions available to choose from. Also, the monitoring target is always centralized, and monitoring is certainly easy to set up and configure. If something breaks down, we know what to look for and where to look for it since only a finite number of components participate in a system, and they have a fairly long life span.

However, microservices are distributed systems and, by nature, more complex than monoliths. So resource utilization and health and performance monitoring are quite essential in a microservice production environment. We can use this diagnostic piece of information to detect...

Instrumentation and telemetry

A monitoring solution is dependent upon instrumentation and telemetry. So it is natural that when we speak about monitoring microservices, we also discuss instrumentation and telemetry data. Logs are nothing but an instrumentation mechanism.

Instrumentation

Now let's see what is instrumentation. Instrumentation is one of the ways through which you can add diagnostic features to applications. It can be formally defined like this:

Most applications will include diagnostic features that generate custom monitoring and debugging information, especially when an error occurs. This is referred to as instrumentation and is usually implemented by adding event and error handling code to the application. -MSDN

Under normal conditions, data from some informational events may not be required, thus reducing the cost of storage and the transactions required to collect it. However, when there is an issue with the application, you have to update the application configuration so that...

The need for monitoring

Microservices are complex, distributed systems. Microservice implementation is the backbone of any modern IT business. Understanding the internals of the services along with their interactions and behaviors will help you make the overall business more flexible and agile. The performance, availability, scale, and security of microservices can directly affect a business and also its revenue. Hence, monitoring microservices is vital. It helps us observe and manage the quality of the service attributes. Let's discuss the scenarios where it is required.

Health monitoring

With health monitoring, we monitor the health of a system and its various components at a certain frequency, typically a few seconds. This ensures that the system and its components behave as expected. With the help of an exhaustive health monitoring system, we can keep tabs on the overall system health, which has data points, such as CPU, memory utilization, and so on. It might be in the form of pings or...

Monitoring challenges

Microservice monitoring presents different challenges. Let's check them out first in this section:

• Scale: One service could be dependent upon the functionality provided by various other microservices. This yields complexity, which is not usual in the case of .NET monolith systems. Instrumenting all these dependencies is quite difficult. Another problem that comes along with scale is the rate of change. With the advancement of continuous deployment and container-based microservices, the code is always in a deployable state. Containers always live for minutes, if not seconds. The same is true for virtual machines. Virtual machines have a life of around a couple of minutes to a couple of hours. In such a case, measuring regular signals, such as CPU usage and memory consumption usage per minute, does not make sense. Container instances sometimes might not be alive for a minute. Before a minute passes, the container instance might have been disposed. This is one of the challenges...

Logging

Logging is a type of instrumentation made available by the system, its various components, or the infrastructure layer. In this section, we will first visit logging challenges and then discuss strategies to reach a solution for these challenges. 

Logging challenges

We will first try to understand the problem with log management in microservices:

• To log the information related to a system event and parameter as well as the infrastructure state, we will need to persist log files. In traditional .NET monoliths, log files are kept on the same machine where the application is deployed. In the case of microservices, they are hosted either on virtual machines or containers. But virtual machines and containers are both ephemeral, which means they do not persist states. In this situation, if we persist log files with virtual machines or containers, we will lose them. This is one of the challenges of log management in microservices.
• In the microservice architecture, there are a number of services...

Monitoring in Azure Cloud

Definitely, there is not even one right-off-the-shelf solution or offering found as of now on Azure or for that matter any cloud provider to the monitoring challenges presented by microservices. Interestingly enough, there are not too many open source tools available that can work with .NET-based microservices.

We are utilizing Microsoft Azure Cloud and cloud services for building our microservices, so it is useful to look for the monitoring capability it comes along with. If you are looking to manage around a couple of hundred microservices, you can utilize a custom monitoring solution (mostly interweaving Powershell scripts) based on a Microsoft Azure-based solution.

We will be primarily focusing on the following logging and monitoring solutions:

• Microsoft Azure Diagnostics: This helps in collecting and analyzing resources through resource and activity logs.
• Application Insights: This helps in collecting all of the telemetry data about our microservices and analyzing...

Other microservice monitoring solutions

Now let's look at some of the popular monitoring solutions that can be used to build a custom microservice monitoring solution. It is inherent that these solutions do not come out of the box; however, they are definitely time tested by the open source community and can be easily integrated within .NET-based environments.

A brief overview of the ELK stack

As we saw, one of the fundamental tools for monitoring is logging. For microservices, there will be a volume of logs generated that are astounding and sometimes not even comprehended by humans. The ELK stack (also referred to as the elastic stack) is the most popular log management platform. It is also a good candidate for microservice monitoring because of its ability to aggregate, analyze, visualize, and monitor. The ELK stack is a toolchain that includes three distinct tools, namely Elasticsearch, Logstash, and Kibana. Let's visit them one by one to understand their role in the ELK stack.

Elasticsearch...

Summary

Debugging and monitoring of microservices is not a simple but a challenging problem. We have used the word challenging on purpose: there is no silver bullet for this. There is no single tool that you can install that works like magic. However, with Azure Diagnostics and Application Insights or with ELK stack or Splunk, you can come up with solutions that will help you solve microservice monitoring challenges. Implementing microservice monitoring strategies, such as application/system monitoring, real user monitoring, synthetic transactions, centralized logging, semantic logging block, and implementation of correlation ID throughout transactional HTTP requests, is a helpful way to monitor microservice implementations.

In the next chapter, we will see how we can scale microservices and the solutions and strategies around scaling microservice solutions.