Jsonnet and Monitoring Mixins

Have you ever tried to template YAML? It’s a nightmare! Most templating languages like Jinja (used by Ansible and Salt), or even Go’s templating language, handle whitespace in generally unintuitive ways to most users. It’s devilishly easy to accidentally add extra space into a template’s output. Combined with YAML’s whitespace sensitivity, it’s a recipe for headaches. And yet – like it or not – YAML is the de-facto configuration language of the present time, used for everything from deploying apps to Kubernetes to configuring your Prometheus servers. So, how do we balance the tedious, error-prone management of large YAML files by hand with the arguably more error-prone desire to template our way out of the repetitiveness?

One solution that has seen relatively broad adoption within the Prometheus community is using a language called Jsonnet to handle generating YAML files. Jsonnet is a superset...

Technical requirements

For this chapter, we’ll be stepping away from our Prometheus deployment for a bit and doing things locally. Consequently, you’ll need these tools installed to follow along:

• jsonnet: https://github.com/google/go-jsonnet/blob/fed90cd9cd733a87f9fb27cfb32a3e08a7695603/README.md#installation-instructions
• jsonnet-bundler: https://github.com/jsonnet-bundler/jsonnet-bundler#install
• promtool: https://github.com/prometheus/prometheus/releases/tag/v2.46.0

The code used in this chapter is available at https://github.com/PacktPublishing/Mastering-Prometheus.

Overview of Jsonnet

Jsonnet was born out of Google by an engineer who wanted to improve upon and extend an internal configuration language known as Google Configuration Language (GCL). It was initially released in 2014 – the same year as Kubernetes. It is truly fortuitous that we received a tool that can generate YAML and the biggest abuser of YAML in the same year. Google giveth and Google taketh away.

At its core, Jsonnet is an extension of JSON – the common, ubiquitous method for sharing data between applications in a human-readable way. This means that any JSON document is also a valid Jsonnet program and will be emitted unchanged when run. Naturally, requiring Jsonnet to be a language that must operate within the constraints of the JSON specification can be limiting, and you will undoubtedly encounter oddities in the language attributable to those constraints.

Jsonnet does not endeavor to be a fully featured programming language as blissful to use as, say,...

Using Jsonnet

By now, we’ve harnessed the power and potential of Jsonnet. But what does it really look like to use it outside of the narrowly-scoped, contrived examples we’ve looked at thus far? How do I use it with Prometheus? This is a Prometheus book, after all. Well, firstly, let’s look at how to use Jsonnet to actually generate files instead of just writing to our terminal.

Generating files

In its most simple form, writing to files with Jsonnet is as easy as providing an additional flag (-o) and file path to the jsonnet CLI. For example, we can write the output of our earlier variables_prometheus.jsonnet file to a new file called out/rules.json by running the following:

$ jsonnet -co ch11/out/rules.json ch11/variables_prometheus.jsonnet

The -c flag is used in addition to -o since -c will create parent directories (such as ch11/out/) for the output file if they don’t exist already. Our ch11/out/rules.json file now contains the expected content...

Monitoring Mixins

The Monitoring Mixins project was started in 2018 and initially driven primarily by two pillars of the Prometheus community, Tom Wilkie (CTO of Grafana Labs) and Frederic Branczyk (of Prometheus Operator fame, among many other things). The stated aim of the project is to “define a minimal standard for how to package together Prometheus alerts, Prometheus recording rules, and Grafana dashboards.”

The project aims to address the needs of two distinct user bases: consumers and providers. In this context, providers are developers and maintainers of software who wish to be able to provide – in a consistent way – a base-level package of configurable Prometheus alerts, recording rules, and Grafana dashboards for their software. This is in line with the thinking that developers of software are the most equipped to be able to identify what metrics their software exposes and which should be of the most interest to its users. On the opposite side...

Summary

In this chapter, we’ve gone from zero to hero in Jsonnet. Take a deep breath and admire the fact that you’ve – most likely – just learned a whole new programming language. With Jsonnet, we can remove some of the tedium of managing YAML files – like those used by Prometheus – by hand. This can help to ensure more consistent and less error-prone management of these files over time by keeping our code DRY.

Not only that, but we’ve learned how the Monitoring Mixin specification enables projects to create reusable, configurable Prometheus rules and Grafana dashboards. By leveraging existing mixins (and/or making our own!), we can ensure greater observability and monitoring of our systems.

In our next chapter, we’ll continue exploring ways to mature our operations of Prometheus monitoring stacks using CI/CD tooling to perform tasks such as validation and testing.

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• Using Jsonnet to Package Together Dashboards, Alerts and Exporters – Tom Wilkie (KubeCon EU 2018): https://www.youtube.com/watch?v=b7-DtFfsL6E
• Taming the Beast: Comparing Jsonnet, Dhall, Cue: https://pv.wtf/posts/taming-the-beast
• Mixins in Object Oriented Programming: https://en.wikipedia.org/wiki/Mixin#History
• Jsonnet project documentation: https://jsonnet.org/
• Monitoring Mixins website: https://monitoring.mixins.dev/
• Monitoring Mixins design proposal: https://github.com/monitoring-mixins/docs/blob/5ef6256185cdc47db8139f576bf463a80d91780e/design.pdf