Running a Prometheus Server

It's time to get our hands on some Prometheus configurations. This chapter will explore the core component of the stack, you will be introduced to common patterns of usage and full setup process scenarios under virtual machines and containers. This will allow you to truly validate the knowledge you've gathered so far and provide you with real examples to test your knowledge.

In brief, the following topics will be covered in this chapter:

• Deep dive into the Prometheus configuration
• Managing Prometheus in a standalone server
• Managing Prometheus in Kubernetes

Deep dive into the Prometheus configuration

One of the key features of Prometheus is, owing to incredibly sane default configurations, that it can scale from a quick test running on a local computer to a production-grade instance, handling millions of samples per second without having to touch almost any of its many knobs and dials. Having said that, it is very useful to know what configuration options are available to be able to get the most value out of Prometheus.

There are two main types of configuration on a Prometheus server—command-line flags and operating logic that provided through configuration files. Command-line flags control the parameters that cannot be changed at runtime, such as the storage path or which TCP port to bind to, and need a full server restart to apply any change done at this level. The configuration files control runtime configuration, such...

Managing Prometheus in a standalone server

As we previously went through several configuration definitions, we're now ready to put them to practice by managing a standalone instance of Prometheus. In these examples, we'll be exposing several configurations while providing an environment to validate them.

Server deploy

To create a new instance of Prometheus, move into the correct repository path, as shown here:

cd chapter05/

Ensure that no other test environments are running and spin up this chapter's environment, like so:

vagrant global-status
vagrant up

After a few moments, the new instance will be available for inspection, and the Prometheus web interface will be accessible at http://192.168.42.10:9090.

...

Managing Prometheus in Kubernetes

Kubernetes is the first project to graduate from the CNCF and is currently the de facto standard for container orchestration. Early on, Heapster was widely used as a monitoring solution that came out-of-the-box with Kubernetes. It started out as a tool to send monitoring data to external systems but then grew to become a monitoring system itself. However, it didn't take long for Prometheus to become the de facto standard monitoring system for Kubernetes clusters. Nowadays, most of the components that make up a Kubernetes cluster have native Prometheus instrumentation.

In the following sections, we'll go into how to integrate Prometheus in a Kubernetes environment by employing examples based on the Kubernetes project and the Prometheus Operator project.

You can find the complete source code of the Kubernetes project and the Prometheus...

Summary

In this chapter, we were introduced to some of the most important configuration concepts for setting up a Prometheus server. This knowledge is fundamental for tailoring Prometheus for your specific scenario. From startup flags to the configuration file, we also spun up an instance to experiment and validate the knowledge we obtained.

As more and more workloads are transitioning to containers, and specifically to Kubernetes, we dived into how to set up and manage Prometheus on such an environment. We began experimenting with static configurations as a stepping stone to understand a more robust approach, the Prometheus Operator.

In the next chapter, we'll go into the most common exporters and build upon what we've learned so that we can successfully collect data from various different sources on Prometheus.

Questions

1. What happens if scrape_timeout is not declared explicitly?
2. How can Prometheus be made to reload its configuration file?
3. How far back does Prometheus look for data before considering a time series stale?
4. What is the difference between relabel_configs and metric_relabel_configs?
5. On the static deployment example, we added a Kubernetes service for the Hey application as a target in Prometheus. What problems will arise if we increase the number of Hey pods?
6. Does static Prometheus static configuration make sense on a Kubernetes environment? Why?
7. In which Kubernetes facilities does the Prometheus Operator rely upon to achieve its goals?

Further reading

• Prometheus configuration: https://prometheus.io/docs/prometheus/latest/configuration/configuration/
• Prometheus TSDB APIs: https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-admin-apis
• Prometheus security: https://prometheus.io/docs/operating/security/
• Kubernetes custom controllers: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#custom-controllers
• Kubernetes custom resources: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions
• Prometheus Operator: https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md