Chapter 7. Monitoring and Metrics

In the previous chapters, we investigated the declarative structures used in Kubernetes objects and resources. With the end goal of having Kubernetes help run software for us, in this chapter we will look at how we can get more information, when we're running our applications at a greater scale, and some open source tools that we can use for that purpose. Kubernetes is already gathering and using some information about how utilized the nodes of the cluster are, and there is a growing capability within Kubernetes to start to collect application-specific metrics, and even use those metrics as a control point for managing the software.

In this chapter, will we dig into these aspects of basic observability and walk through how you can set them up for your local development use, and how to leverage them to gather, aggregate, and expose details of how your software is running, when you scale it up. Topics within this chapter will include:

• Built-in metrics with Kubernetes...

Built-in metrics with Kubernetes

Kubernetes comes built in with some basic instrumentation to know how much CPU and memory are consumed on each node in the cluster. Exactly what is captured and how it is captured has been evolving rapidly in recent Kubernetes releases (1.5 through 1.9). Many Kubernetes installations will be capturing information about what resources the underlying containers are using with a program called cAdvisor. This code was created by Google to collect, aggregate, and expose the metrics of how containers are operating, as a critical step of being able to know where best to place new containers, based on what resources a node has and where resources are available.

Every node within a Kubernetes cluster will have cAdvisor running and collecting information, and this, in turn, is captured and used by kubelet, which is the local agent on every node that is responsible for starting, stopping, and managing the various resources needed to run containers.

cAdvisor exposes a...

Kubernetes concept – Quality of Service

When a pod is created in Kubernetes, it is also assigned a Quality of Service class, based on the data provided about the pod when it was requested. This is used by the scheduler to provide some upfront assurances during the scheduling process, and later in the management of the pods themselves. The three classes supported are:

• Guaranteed
• Burstable
• BestEffort

Which class is assigned to your pod is based on what resource limits and requests you report with the containers within your pod for CPU and memory utilization. In the previous examples, none of the containers were assigned a requests or limit, so all of those pods were classified as BestEffort when they were run.

Resource requests and limits are defined on each container within a pod. If we add a request to a container, we are asking for Kubernetes to make sure that the cluster has sufficient resources to run our pod (memory, CPU, or both) and it will validate that availability as a part of the scheduling...

Capturing metrics with Prometheus

Prometheus is a prominent open source tool used for monitoring, and quite a bit of symbiotic work is happening between it and the Kubernetes community. Kubernetes application metrics are exposed in the Prometheus format. This format includes the data types of counter, gauge, histogram, and summary, as well as a means of specifying labels to be associated with specific metrics. As Prometheus and Kubernetes have both evolved, the metrics format from Prometheus appears to be emerging as a de facto standard within the project and across its various components.

More information about this format is available online at the Prometheus project's documentation:

• https://prometheus.io/docs/concepts/data_model/
• https://prometheus.io/docs/concepts/metric_types/
• https://prometheus.io/docs/instrumenting/exposition_formats/

Beyond the metrics format, Prometheus offers quite a variety of capabilities as its own open source project, and is used outside Kubernetes. The architecture...

Installing Grafana

Grafana isn't by itself a complex installation, but configuring it can be. Grafana can plug into a number of different backend systems and provide dashboarding and graphing for them. In our example, we would like to have it provide dashboards from Prometheus. We will set up an installation and then configure it through its user interface.

We can use Helm again to install Grafana, and since we have put Prometheus in the namespace monitoring, we will do the same with Grafana. We could do helm fetch and install to look at the charts. In this case, we will just install them directly:

helm install stable/grafana -n viz --namespace monitoring

In the resulting output, you will see a secret, ConfigMap, and deployment among the resources created, and in the notes, something like:

NOTES:
1. Get your 'admin' user password by running:

kubectl get secret --namespace monitoring viz-grafana -o jsonpath="{.data.grafana-admin-password}" | base64 --decode ; echo

2. The Grafana server can...

Using Prometheus to view application metrics

While you could add a job within Prometheus to include the configuration to scrape Prometheus metrics from a specific endpoint, the installation we did earlier includes a configuration that will update what it is looking at dynamically, based on annotations on pods. One of the benefits of Prometheus is that it has support for automatically detecting changes in your cluster, based on annotations, and it can look up the endpoints for the pods that back a service.

Since we deployed Prometheus using Helm, you can find the relevant configuration embedded within the values.yaml file. Look for the Prometheus job kubernetes-service-endpoints, and you will find both the configuration and some documentation of how it can be used. If you don't have the files locally, you can view this configuration at https://github.com/kubernetes/charts/blob/master/stable/prometheus/values.yaml#L747-L776.

This configuration looks for services within the cluster that have...

Summary

In this chapter, we introduced Prometheus and showed how to install it, used it to capture metrics from your Kubernetes cluster, and showed how to install and use Grafana to provide dashboards, using the metrics captured and temporarily stored in Prometheus. We then looked at how you can expose custom metrics from your own code and leverage Prometheus to capture them, along with a few examples of metrics you might be interested in tracking, such as error rate, response time, and throughput.

In the next chapter, we continue to look into observability for our applications with tools to help us capture logs and traces.