Chapter 3: Auto-Instrumentation

The purpose of telemetry is to give people information about systems. This data is used to make informed decisions about ways to improve software and prevent disasters from occurring. In the case of an outage, analytics tools can help us investigate the root cause of the interruption by interpreting telemetry. Once the event has been resolved, the recorded traces, metrics, and logs can be correlated retroactively to gain a complete picture of what happened. In all these cases, the knowledge that's gained from telemetry assists in solving problems, be it future, present, or past, in applications within an organization. Being able to see the code is very rarely the bread and butter of an organization, which sometimes makes conversations about investing in observability difficult. Decision-makers must constantly make tradeoffs regarding where to invest. The upfront cost of instrumenting code can be a deterrent to even getting started, especially if...

Technical requirements

The application in this chapter simulates the broken telephone game. If you're not familiar with this game, it is played by having one person think of a phrase and whisper it to the second player. The second player listens to the best of their ability and whispers it to the third player; this continues until the last player shares the message they received with the rest of the group.

Each application represents a player, with the first one printing out the message it is sending, then placing the message in a request object that's sent to the next application. The last application in the game will print out the message it receives. The following diagram shows the data flow of requests and responses through the system:

Figure 3.1 – Architectural diagram of the example application

The communication between each service is done via gRPC (https://grpc.io), a remote procedure call system developed by Google. For the sake...

What is auto-instrumentation?

In the very early days of the OpenTelemetry project, a proposal was created to support producing telemetry without manual instrumentation. As we mentioned earlier in this book, OpenTelemetry uses OpenTelemetry Enhancement Proposals or OTEPs to propose significant changes or new work before producing a specification. One of the very first OTEPs to be produced by the project (https://github.com/open-telemetry/oteps/blob/main/text/0001-telemetry-without-manual-instrumentation.md) described the need to support users that wanted to produce telemetry without having to modify the code to do so:

Cross-language requirements for automated approaches to extracting portable telemetry data with zero source code modification. – OpenTelemetry Enhancement Proposal #0001

Being able to get started with OpenTelemetry with very little effort for new users was very much a goal from the start of the project. The hope was to address one of the pain points of producing...

Bytecode manipulation

The Java implementation of auto-instrumentation for OpenTelemetry leverages the Java Instrumentation API to instrument code (https://docs.oracle.com/javase/8/docs/api/java/lang/instrument/Instrumentation.html). This API is defined as part of the Java language and can be used by anyone interested in collecting information about an application.

OpenTelemetry Java agent

The OpenTelemetry Java agent is distributed to users via a single Java archive (JAR) file, which can be downloaded from the opentelemetry-java-instrumentation repository (https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases). The JAR contains the following components:

• The javaagent module. This is called by the Java Instrumentation API.
• Instrumenting libraries for various frameworks and third-party libraries.
• The tooling to initialize and configure the OpenTelemetry components. These will be used to produce telemetry and deliver it to its destination...

Runtime hooks and monkey patching

In Python, unlike in Java, where a single archive contains everything that's needed to support auto-instrumentation, the implementation relies on several separate components that must be discussed to help us fully understand how auto-instrumentation works.

Instrumenting libraries

Instrumentation libraries in Python rely on one of two mechanisms to instrument third-party libraries:

• Event hooks are exposed by the libraries being instrumented, allowing the instrumenting libraries to register and produce telemetry as events occur.
• Any intercepting calls to libraries are instrumented and are replaced at runtime via a technique known as monkey patching (https://en.wikipedia.org/wiki/Monkey_patch). The instrumenting library receives the original call, produces telemetry data, and then calls the underlying library.

Monkey patching is like bytecode injection in that the applications make calls to libraries without suspecting that...

Summary

With auto-instrumentation, it's possible to reduce the time that's required to instrument an existing application. Reducing the friction to get started with telemetry gives users a chance to try it before investing significant amounts of time in manual instrumentation. And although the data that's generated via auto-instrumentation is likely not enough to get to the bottom of issues in complex systems, it's a solid starting point. Auto-instrumentation can also be quite useful when you're instrumenting an unfamiliar system.

The use of instrumentation libraries allows users to gain insight into what the libraries they're using are doing, without having to learn the ins and outs of them. The OpenTelemetry libraries that are available at the time of writing can be used to instrument existing code by following the online documentation that's been made available by each language. As we'll learn in Chapter 7, Instrumentation Libraries, using...