Monitoring and Logging

Monitoring and logging are a crucial part of a site's reliability. We've learned how to leverage various controllers to take care of our application, and about utilizing service together with Ingress to serve our web applications. Next, in this chapter, we'll learn how to keep track of our application by means of the following topics:

• Getting status snapshot of a container
• Monitoring in Kubernetes
• Converging metrics from Kubernetes by Prometheus
• Concepts of logging in Kubernetes
• Logging with Fluentd and Elasticsearch

Inspecting a container

Whenever our application behaves abnormally, we will definitely want to know what happened, using all means, such as checking logs, resource usage, processes watchdog, or even getting into the running host directly to dig problems out. In Kubernetes, we have kubectl get and kubectl describe that can query deployment states, which will help us determine if an application has crashed or works as desired.

Further, if we want to know what is going on from the outputs of an application, we also have kubectl logs that redirects a container's stdout to our Terminal. For CPU and memory usage stats, there's also a top-like command we can employ, kubectl top. kubectl top node, which gives an overview of the resource usages of nodes, and kubectl top pod <POD_NAME> which displays per-pod usage:

# kubectl top node
NAME        CPU(cores)   CPU%      MEMORY...

Monitoring in Kubernetes

Since we now know how to examine our applications in Kubernetes, it's quite natural that we should have a mechanism to do so constantly to detect any incident at the first occurrence. To put it another way, we need a monitoring system. A monitoring system collects metrics from various sources, stores and analyzes data received, and then responds to exceptions. In a classical setup of application monitoring, we would gather metrics from, at the very least, three different layers of our infrastructure to ensure our service's availability as well as quality.

Application

The data we're concerned with at this level involves the internal states of an application, which can help us determine...

Hands-on monitoring

So far, we've learned lots of principles to fabricate an impervious monitoring system in Kubernetes toward a robust service, and it's time to implement a pragmatic one. Because the vast majority of Kubernetes components expose their instrumented metrics on a conventional path in Prometheus format, we are free to use any monitoring tool with which we are acquainted as long as the tool understands the format. In this section, we'll set up an example with an open-source project, Prometheus (https://prometheus.io), which is a platform-independent monitoring tool. Its popularity in Kubernetes' ecosystem is for not only its powerfulness but also for its being backed by the Cloud Native Computing Foundation (https://www.cncf.io/), who also sponsors the Kubernetes project.

...

Logging events

Monitoring with quantitative time series of a system status enables us to briskly dig out which components in our system failed, but it's still inadequate to diagnose with the root cause under syndromes. As a result, a logging system that gathers, persists, and searches logs is certainly helpful for uncovering the reason why something went wrong by means of correlating events with the anomalies detected.

In general, there are two main components in a logging system: the logging agent and the logging backend. The former is an abstract layer to a program. It gathers, transforms, and dispatches logs to the logging backend. A logging backend warehouses all logs received. As with monitoring, the most challenging part of building a logging system for Kubernetes is ascertaining how to gather logs from containers to a centralized logging backend. Typically, there are...

Extracting metrics from logs

The monitoring and logging system we built around our application on top of Kubernetes is shown in the following diagram:

The logging part and the monitoring part look like two independent tracks, but the value of the logs is much more than a collection of short texts. They are structured data and emitted with timestamps as usual; as such, the idea to transform logs into time-series data is promising. However, although Prometheus is extremely good at processing time-series data, it cannot ingest texts without any transformation.

An access log entry from HTTPD looks like this:

10.1.8.10 - - [07/Jul/2017:16:47:12 0000] "GET /ping HTTP/1.1" 200 68.

It consists of the request IP address, time, method, handler, and so on. If we demarcate log segments by their meanings, counted sections can then be regarded as a metric sample like this: "...

Summary

At the start of this chapter, we described how to get the status of running containers quickly by means of built-in functions such as kubectl. Then we expanded the discussion to concepts and principles of monitoring, including why it is necessary to do monitoring, what to monitor, and how to monitor. Afterwards, we built a monitoring system with Prometheus as the core, and set up exporters to collecting metrics from Kubernetes. The fundamentals of Prometheus were also introduced so that we can leverage metrics to gain more understanding of our cluster as well as the applications running inside. On the logging part, we mentioned common patterns of logging and how to deal with them in Kubernetes, and deployed an EFK stack to converge logs. The system we built in this chapter facilitates the reliability of our service. Next, we are advancing to set up a pipeline to deliver...