Choosing the Right Service Discovery

When tackling dynamic environments, manually maintaining a file of targets is not an option. Service discovery handles the complexity of an ever-changing infrastructure for you, making sure that no service or host slips through the cracks. This chapter focuses on how to take advantage of Prometheus service discovery to decrease the infrastructure management toil regarding coping with constant change.

In brief, the following topics will be covered in this chapter:

• Test environment for this chapter
• Running through the service discovery options
• Using a built-in service discovery
• Building a custom service discovery

Test environment for this chapter

In this chapter, we'll be focusing on service discovery. For this, we'll be deploying two new instances to simulate a scenario where Prometheus generates targets dynamically using a popular service discovery software. This approach will allow us to not only expose the required configurations, but also validate how everything works together.

The setup we'll be using resembles the following diagram:

Figure 12.1: Test environment for this chapter

The usual deployment pattern for Consul is to have an agent running in client mode on every node in the infrastructure, which will then contact Consul instances running in server mode. Furthermore, client instances act as API proxies, so it is common practice for Prometheus Consul service discovery to be configured using the localhost. However, to make their different responsibilities clear...

Running through the service discovery options

Prometheus comes with several discovery integrations available out of the box. These cover most of the mainstream data sources for application and machine inventories, such as public and private cloud compute APIs, VM and container orchestration systems, standalone service registration and discovery systems, among others. For those discovery mechanisms that aren't directly supported by Prometheus, integration can be done through a generic discovery system using the filesystem and some glue code, as we'll see later in this chapter.

Every integration works in the same way – by setting all the discovered addresses as targets and their associated metadata as temporary labels (not persisted without some relabeling to keep them). For each discovered target, the __address__ label is usually set to the service address and...

Using a built-in service discovery

To understand how the integration between Prometheus and a service discovery provider works, we're going to rely on our test environment. Going even further, we'll provide a working example of Prometheus running in Kubernetes, relying on its native service discovery for this platform. These hands-on examples will showcase how everything ties together, helping you figure out not only the benefits but, above all, the simplicity of these mechanics.

Using Consul service discovery

For this chapter, we configured Consul as an example service discovery system in our virtual machine-based test environment – Consul is quite simple to set up, which makes it perfect for our example...

Building a custom service discovery

Even with all the available service discovery options, there are numerous other systems/providers that are not supported out of the box. For those cases, we've got a couple of options:

• Open a feature request for Prometheus to support that particular service discovery, and rely on the community and/or maintainers to implement it.
• Implement the service discovery integration yourself in Prometheus and either maintain a fork or contribute it back to the project.
• Figure out a way to get the targets you require into your Prometheus instances with minimal maintenance work and time cost, and without relying on the Prometheus roadmap to get the job done.

The first two options aren't great, as they are either outside of our control or are cumbersome to maintain. Furthermore, adding additional service discovery integrations to Prometheus without...

Summary

In this chapter, we had the opportunity to understand why service discovery is essential for managing ever-growing infrastructure in a sane way. Prometheus leverages several service discovery options out of the box, which can kick-start your adoption in a very quick and friendly manner. We went through the available options Prometheus provides for service discovery, and showed you what to expect from them. We then stepped into a couple of examples using Consul and Kubernetes to materialize the concepts we exposed previously. Finally, we went through how to integrate a custom service discovery with Prometheus by using the recommended approach and relying on file_sd.

In the next chapter, we'll go through how to scale and federate Prometheus.

Questions

1. Why should you use a service discovery mechanism in Prometheus?
2. When you're using a cloud provider service discovery, what is the major requirement for setting the integration?
3. What are the types of records supported by the DNS-based service discovery integration?
4. What purpose does the concept of role serve in the Kubernetes service discovery integration?

5. When you're building a custom service discovery, what available integration will you be relying upon?
6. Do you need to reload Prometheus when a target file configured in file_sd is updated?
7. What is the recommended way of building your own custom service discovery?

Further reading

• Prometheus service discovery configuration: https://prometheus.io/docs/prometheus/latest/configuration/configuration