Extending HorizontalPodAutoscaler with Custom Metrics

Computers make excellent and efficient servants, but I have no wish to serve under them.

- Spock

Adoption of HorizontalPodAutoscaler (HPA) usually goes through three phases.

The first phase is discovery. The first time we find out what it does, we usually end up utterly amazed. "Look at this. It scales our applications automatically. I don't need to worry about the number of replicas anymore."

The second phase is the usage. Once we start using HPA, we quickly realize that scaling applications based memory and CPU is not enough. Some apps do increase their memory and CPU usage with the increase in load, while many others don't. Or, to be more precise, not proportionally. HPA, for some applications, works well. For many others, it does not work at all, or it is not enough. Sooner or later, we'll need...

Creating a cluster

The vfarcic/k8s-specs (https://github.com/vfarcic/k8s-specs) repository will continue to serve as our source of Kubernetes definitions. We'll make sure that it is up-to-date by pulling the latest version.

All the commands from this chapter are available in the 05-hpa-custom-metrics.sh (https://gist.github.com/vfarcic/cc546f81e060e4f5fc5661e4fa003af7) Gist.

 1  cd k8s-specs
 2
 3  git pull

The requirements are the same as those we had in the previous chapter. The only exception is EKS. We'll continue using the same Gists as before for all other Kuberentes flavors.

A note to EKS users
Even though three t2.small nodes we used so far have more than enough memory and CPU, they might not be able to host all the Pods we'll create. EKS (by default) uses AWS networking. A t2.small instance can have a maximum of three network interfaces, with four IPv4 address...

Using HorizontalPodAutoscaler without metrics adapter

If we do not create a metrics adapter, Metrics Aggregator only knows about CPU and memory usage related to containers and nodes. To make things more complicated, that information is limited only to the last few minutes. Since HPA is just concerned about Pods and containers inside them, we are limited to only two metrics. When we create an HPA, it will scale or de-scale our Pods if memory or CPU consumption of the containers that constitute those Pods is above or below predefined thresholds.

Metrics Server periodically fetches information (CPU and memory) from Kubelets running inside worker nodes.

Those metrics are passed to Metrics Aggregator which, in this scenario, does not add any additional value. From there on, HPAs periodically consult the data in the Metrics Aggregator (through its API endpoint). When there is a discrepancy...

Exploring Prometheus Adapter

Given that we want to extend the metrics available through the Metrics API, and that Kubernetes allows us that through it's Custom Metrics API (https://github.com/kubernetes/metrics/tree/master/pkg/apis/custom_metrics), one option to accomplish our goals could be to create our own adapter.

Depending on the application (DB) where we store metrics, that might be a good option. But, given that it is pointless to reinvent the wheel, our first step should be to search for a solution. If someone already created an adapter that suits our needs, it would make sense to adopt it, instead of creating a new one by ourselves. Even if we do choose something that provides only part of the features we're looking for, it's easier to build on top of it (and contribute back to the project), than to start from scratch.

Given that our metrics are stored...

Creating HorizontalPodAutoscaler with custom metrics

As you already saw, Prometheus Adapter comes with a set of default rules that provide many metrics that we do not need, and not all those that we do. It's wasting CPU and memory by doing too much, but not enough. We'll explore how we can customize the adapter with our own rules. Our next goal is to make the adapter retrieve only the nginx_ingress_controller_requests metric since that's the only one we need. On top of that, it should provide that metric in two forms. First, it should retrieve the rate, grouped by the resource.

The second form should be the same as the first but divided with the number of replicas of the Deployment that hosts the Pods where Ingress forwards the resources.

That one should give us an average number of requests per replica and will be a good candidate for our first HPA definition...

Combining Metric Server data with custom metrics

So far, the few HPA examples used a single custom metric to decide whether to scale the Deployment. You already know from the Chapter 1, Autoscaling Deployments and StatefulSets Based on Resource Usage, that we can combine multiple metrics in an HPA. However, all the examples in that chapter used data from the Metrics Server. We learned that in many cases memory and CPU metrics from the Metrics Server are not enough, so we introduced Prometheus Adapter that feeds custom metrics to the Metrics Aggregator. We successfully configured an HPA to use those custom metrics. Still, more often than not, we'll need a combination of both types of metrics in our HPA definitions. While memory and CPU metrics are not enough by themselves, they are still essential. Can we combine both?

Let's take a look at yet another HPA definition...

The complete HorizontalPodAutoscaler flow of events

Metrics Server is fetching memory and CPU data from Kubelets running on the worker nodes. In parallel, Prometheus Adapter is fetching data from Prometheus Server which, as you already know, pulls data from different sources. Data from both Metrics Server and Prometheus Adapter is combined in Metrics Aggregator.

HPA is periodically evaluating metrics defined as scaling criteria. It's fetching data from Metrics Aggregator, and it does not really care whether they're coming from Metrics Server, Prometheus Adapter, or any other tool we could have used.

Once scaling criteria is met, HPA manipulates Deployments and StatefulSets by changing their number of replicas.

As a result, rolling updates are performed by creating and updating ReplicaSets which, in turn, create or remove Pods.

Figure 5-3: HPA using a combination of...

Reaching nirvana

Now that we know how to add almost any metric to HPAs, they are much more useful than what it seemed in the Chapter 1, Autoscaling Deployments and StatefulSets Based on Resource Usage. Initially, HPAs weren't very practical since memory and CPU are, in many cases, insufficient for making decisions on whether to scale our Pods. We had to learn how to collect metrics (we used Prometheus Server for that), and how to instrument our applications to gain more detailed visibility. Custom Metrics was the missing piece of the puzzle. If we extend the "standard" metrics (CPU and memory) with the additional metrics we need (for example, Prometheus Adapter), we gain a potent process that will keep the number of replicas of our applications in sync with internal and external demands. Assuming that our applications are scalable, we can guarantee that they will...

What now?

Please note that we used autoscaling/v2beta1 version of HorizontalPodAutoscaler. At the time of this writing (November 2018), only v1 is stable and production-ready. However, v1 is so limited (it can use only CPU metrics) that it's almost useless. Kubernetes community worked on new (v2) HPA for a while and, in my experience, it works reasonably well. The main problem is not stability but potential changes in the API that might not be backward compatible. A short while ago, autoscaling/v2beta2 was released, and it uses a different API. I did not include it in the book because (at the time of this writing) most Kubernetes clusters do not yet support it. If you're running Kubernetes 1.11+, you might want to switch to v2beta2. If you do so, remember that you'll need to make a few changes to the HPA definitions we explored. The logic is still the same, and...