In the previous chapter, we looked into the theory behind end-to-end tracing, and various architectural decisions one must make when building a distributed tracing infrastructure, including which data formats can be used for propagating metadata between processes and for exporting tracing data to a tracing backend. Fortunately, as we will see in this chapter, an end user of a tracing infrastructure, someone who wants to instrument their business application, or their open source framework, or library, typically does not need to worry about those decisions.

We only briefly touched upon the notion of instrumentation and trace points before, so in this chapter, we will dive deep into the question of instrumentation, using three canonical "Hello, World!" applications in Go, Java, and Python. You may be having Jules Winnfield's reflex right now: "Say Hello, World! again," but I promise to make it interesting...

In Chapter 4, Instrumentation Basics with OpenTracing, we reviewed the basics of instrumenting a microservices-based application for distributed tracing using the OpenTracing APIs. If you went through all the exercises, you deserve a medal! The Hello application was intentionally very simple and involved only blocking synchronous calls between microservices.

In this chapter, we will attempt to instrument an online chat application, Tracing Talk, which uses asynchronous messaging-based interactions between microservices built on top of Apache Kafka. We will see how metadata context can be passed through messaging systems using the same OpenTracing primitives we already discussed, and how causal relationships between spans can be modeled differently than in the plain RPC scenarios.

We will continue using the OpenTracing API, even though the same instrumentation principles would apply to other tracing APIs, such as Zipkin...

In the microservices-ruled world, end-to-end tracing is no longer a "nice-to-have" feature, but "table stakes" for understanding modern cloud-native applications. In Chapter 4, Instrumentation Basics with OpenTracing, we saw an example of what it takes to manually instrument a simple "Hello, world!" application for distributed tracing. If, after reading it, you were left with the impression, "oh, that is a lot of work," then it achieved one of its goals. Developing and deploying instrumentation in order to get high-quality tracing data is absolutely the largest, but not the only, challenge of rolling out a distributed tracing solution in an organization. In Chapter 13, Implementing Tracing in Large Organizations, we will review some practical techniques for making the process easier from an organizational point of view.

In this chapter, we will discuss:

In previous chapters, we discussed multiple methods of extracting tracing data from applications, either by adding instrumentation directly to the application code, or enabling the instrumentation dynamically at runtime through configuration. I also mentioned agent-based instrumentation, often provided by commercial APM vendors, which works by injecting trace points externally into the customer's program, using techniques such as monkey-patching and bytecode manipulation. All these methods can be classified as white-box instrumentation, since they all require modification of the application's code, either explicitly or implicitly at runtime. In Chapter 3, Distributed Tracing Fundamentals, we also discussed black-box techniques that work purely by correlating externally observed telemetry, such as logs used by the Mystery Machine [1].

In this chapter, we will discuss and try in practice how service meshes...

The gathering of monitoring data in production is always a compromise between the costs, in terms of storage and performance overhead, and the expressiveness of the collected data. The more data we collect, the better we hope to be able to diagnose the situation, should something go wrong, yet we don't want to slow down the applications or pay exorbitant bills for storage. Even though most logging frameworks support multiple levels of log severity, a common wisdom is to tune the loggers in production to discard anything logged with the debug level or lower. Many organizations even adopt the rule that successful requests should leave no logs at all, and you only log when there is some issue with the request.

Distributed tracing is not immune to this compromise either. Depending on the verbosity of the instrumentation, tracing data can easily exceed the volume of the actual business traffic sustained by an application. Collecting all that data in...