Integrating Prometheus with OpenTelemetry

In my career, I have never encountered an observability project on the level of OpenTelemetry. Not since the introduction of SNMP has there been such a widely adopted effort to standardize around a data format and ecosystem for application and system telemetry.

To many a cynical modern mind, the concepts of vendor agnosticism and unifying standards may seem like a shining ideal whose light dwindled sometime in the late 20th century after common Internet standards were established around things such as HTTP and DNS. Sure, there are updates and iterations on those things, but what entirely new technology standard really sees widespread adoption? So much modern innovation is proprietary and vendor-specific, perpetuating cycles of vendor lock-in. Yet, some of the worst perpetrators of vendor lock-in – cloud observability and application performance monitoring (APM) vendors – have forsaken their proprietary protocols and data formats...

Technical requirements

For this chapter, we’ll be leveraging our Prometheus environment built in Chapter 2. In addition to that, you’ll need these tools installed to follow along:

• OpenTelemetry Collector (otelcol; not otelcol-contrib):https://github.com/open-telemetry/opentelemetry-collector-releases/releases/tag/v0.92.0

The code used in this chapter is available at https://github.com/PacktPublishing/Mastering-Prometheus.

Introducing OpenTelemetry

At its core, OpenTelemetry (commonly abbreviated as OTel) is not a tangible technology such as Prometheus, Thanos, or Kubernetes. It’s not something that you “run,” per se. Instead, OpenTelemetry is a specification.

The OpenTelemetry project itself was born out of a rare event: the consolidation of two competing open source projects. Before OpenTelemetry became a project, two competing open source standards sought to address the problem of vendor lock-in in the observability space: OpenCensus and OpenTracing.

OpenCensus, which began in 2017, was the open source solution sponsored by Google and covered both tracing and metrics. OpenTracing, which began in 2016, was a CNCF project (like Prometheus and Kubernetes) and was – as its name implies – primarily focused on tracing. However, as one of my favorite XKCD comics explains, multiple competing standards are less than ideal.

Figure 14.1 –...

Collecting Prometheus metrics with the OpenTelemetry collector

The first step in integrating the OpenTelemetry collector with Prometheus is to get metric data. There are myriad ways to get metric data into the OpenTelemetry collector, but we’ll operate based on the assumption that we’re collecting data from things that already expose Prometheus metrics. In order to collect data from those applications, we can use the prometheus receiver from the OpenTelemetry collector contrib repository.

The prometheus receiver supports all of the same configuration settings that Prometheus itself supports within its scrape_configs section (including service discovery!). Consequently, if you’ve made it this far, it should be easy to configure.

To keep things simple, we’ll just configure the OpenTelemetry collector to scrape itself. Rather than sending data to Prometheus just yet, we’ll use the debug exporter to output data directly to our terminal instead.

...

Sending metrics to Prometheus with the OpenTelemetry collector

At the time of writing, sending data to Prometheus over OTLP is still considered an experimental feature. In fact, the functionality came out during the process of writing this book (in Prometheus version 2.47.0), so we’ll need to upgrade the version of Prometheus we’ve been running so far in order to enable support for it.

Configuring Prometheus

To support the OTLP receiver on Prometheus, we need to adjust our Helm chart to install a newer version of Prometheus and add a --enable-feature=otlp-write-receiver flag to the Prometheus process. Additionally, to avoid any complications inherent in pushing data to multiple Prometheus replicas, we’ll scale back down to a single replica.

The Helm values file we’ll use looks like the following:

grafana:
  enabled: true
  defaultDashboardsTimezone: browser
  adminUser: root
  adminPassword: m@ster1ngPr0m3th3us...

Summary

In this chapter, we learned about the OpenTelemetry project and the exciting prospects it poses for the future of observability. Rather than superseding Prometheus or rendering it obsolete, we saw how the two are able to natively integrate and provide added functionality.

Importantly, we saw that OpenTelemetry’s integration with Prometheus should be used with intentionality and not necessarily as a wholesale replacement of Prometheus’s default, direct pull-based model. Nevertheless, OpenTelemetry is an exciting, rapidly developing project that you should keep your eye on for years to come as its integrations develop and improve.

In our next and final chapter, we’ll look at what you can do right now to make your observability environment as robust as possible. Remember, way back in Chapter 1, when we talked about how Prometheus is just one piece in a larger picture of making a system observable? In our next chapter, we’ll look at what the other...

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• OpenTelemetry specification: https://opentelemetry.io/docs/specs/otel/
• OpenTelemetry line protocol specification: https://opentelemetry.io/docs/specs/otlp/
• OpenTelemetry semantic conventions: https://opentelemetry.io/docs/specs/semconv/
• Recommended books on OpenTelemetry:
  • Boten, Alex. Cloud-Native Observability With OpenTelemetry: Learn to Gain Visibility Into Systems by Combining Tracing, Metrics, and Logging With OpenTelemetry. Packt Publishing, 2022.
  • Parker, Austin, and Ted Young. Learning OpenTelemetry: Setting up and Operating a Modern Observability System. O’Reilly Media, 2024.
  • Majors, Charity, et al. Observability Engineering. O’Reilly Media, 2022.