Monitoring and Logging

Monitoring and logging are crucial parts of a site's reliability. So far, we've learned how to use various controllers to take care of our application. We have also looked at how to utilize services together with Ingress to serve our web applications, both internally and externally. In this chapter, we'll gain more visibility over our applications by looking at the following topics:

• Getting a status snapshot of a container
• Monitoring in Kubernetes
• Converging metrics from Kubernetes with Prometheus
• Various concepts to do with logging in Kubernetes
• Logging with Fluentd and Elasticsearch
• Gaining insights into traffic between services using Istio

Inspecting a container

Whenever our application behaves abnormally, we need to figure out what has happened with our system. We can do this by checking logs, resource usage, a watchdog, or even getting into the running host directly to dig out problems. In Kubernetes, we have kubectl get and kubectl describe, which can query controller states about our deployments. This helps us determine whether an application has crashed or whether it is working as desired.

If we want to know what is going on using the output of our application, we also have kubectl logs, which redirects a container's stdout and stderr to our Terminal. For CPU and memory usage stats, there's also a top-like command we can employ, which is kubectl top. kubectl top node gives an overview of the resource usage of nodes, while kubectl top pod <POD_NAME> displays per-pod usage:

$ kubectl top node...

Monitoring in Kubernetes

We now know how to examine our applications in Kubernetes. However, we are not yet confident enough to answer more complex questions, such as how healthy our application is, what changes have been made to the CPU usage from the new patch, when our databases will run out of capacity, and why our site rejects any requests. We therefore need a monitoring system to collect metrics from various sources, store and analyze the data received, and then respond to exceptions. In a classical setup of a monitoring system, we would gather metrics from at least three different sources to measure our service's availability, as well as its quality.

Monitoring applications

The data we are concerned with relates...

Hands-on monitoring

So far, we've learned about a wide range of principles that are required to create an impervious monitoring system in Kubernetes, which allows us to build a robust service. It's time to implement one. Because the vast majority of Kubernetes components expose their instrumented metrics on a conventional path in Prometheus format, we are free to use any monitoring tool with which we are acquainted, as long as the tool understands the format. In this section, we'll set up an example with Prometheus. Its popularity in the Kubernetes ecosystem is not only due to its power, but also for its backing by the Cloud Native Computing Foundation (https://www.cncf.io/), which also sponsors the Kubernetes project.

Getting to know Prometheus

...

Logging events

Monitoring with a quantitative time series of the system status enables us to quickly identify which components in our system have failed, but it still isn't capable of diagnosing the root cause of a problem. What we need is a logging system that gathers, persists, and searches logs, by means of correlating events with the anomalies detected. Surely, in addition to troubleshooting and postmortem analysis of system failures, there are also various business use cases that need a logging system.

In general, there are two main components in a logging system: the logging agent and the logging backend. The former is an abstract layer of a program. It gathers, transforms, and dispatches logs to the logging backend. A logging backend warehouses all logs received. As with monitoring, the most challenging part of building a logging system for Kubernetes is determining...

Incorporating data from Istio

In a service mesh, the gateway between every service is the front proxy. For this reason, the front proxy is, unsurprisingly, a rich information source for things running inside the mesh. However, if our tech stack already has similar components, such as load balancers or reverse proxies for internal services, then what's the difference between collecting traffic data from them and the service mesh proxy? Let's consider the classical setup:

SVC-A and SVC-B make requests to SVC-C. The data gathered from the load balancer for SVC-C represents the quality of SVC-C. However, as we don't have any visibility over the path from the clients to SVC-C, the only way to measure the quality between SVC-A or SVC-B and SVC-C is either by relying on a mechanism built on the client side, or by putting probes in the network that the clients are in....

Summary

At the start of this chapter, we described how to get the status of running containers quickly by means of built-in functions such as kubectl. Then, we expanded the discussion to look at the concepts and principles of monitoring, including why, what, and how to monitor our application on Kubernetes. Afterward, we built a monitoring system with Prometheus as the core, and set up exporters to collect metrics from our application, system components, and Kubernetes units. The fundamentals of Prometheus, such as its architecture and query domain-specific language were also introduced, so we can now use metrics to gain insights into our cluster, as well as the applications running inside, to not only retrospectively troubleshoot, but also detect potential failures. After that, we described common logging patterns and how to deal with them in Kubernetes, and deployed an EFK stack...