Chapter 2: OpenTelemetry Signals – Traces, Metrics, and Logs

Learning how first to instrument an application can be a daunting task. There's a fair amount of terminology to understand before jumping into the code. I always find that seeing the finish line helps me get motivated and stay on track. This chapter's goal is to see what telemetry generated by OpenTelemetry looks like in practice while learning about the theory. In this chapter, we will dive into the specifics of the following:

• Distributed tracing
• Metrics
• Logs
• Producing consistent quality data with semantic conventions

To help us get a more practical sense of the terminology and get comfortable with telemetry, we will look at the data using various open source tools that can help us to query and visualize telemetry.

Technical requirements

This chapter will use an application that is already instrumented with OpenTelemetry, a grocery store, and several backends to walk through the different concepts of the signals. The environment we will be launching relies on Docker Compose. The first step is to install Docker by following the installation instructions at https://docs.docker.com/get-docker/. Ensure Docker is running on your local system by using the following command:

$ docker version
Client:
 Cloud integration: 1.0.14
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16.3 ...

Next, let's ensure Compose is also installed by running the following command:

$ docker compose version
Docker Compose version 2.0.0-beta.1

Important Note

Compose was added to the Docker client in more recent client versions. If the previous command...

Traces

Distributed tracing is the foundation behind the tracing signal of OpenTelemetry. A distributed trace is a series of event data generated at various points throughout a system tied together via a unique identifier. This identifier is propagated across all components responsible for any operation required to complete the request, allowing each operation to associate the event data to the originating request. The following diagram gives us a simplified example of what a single request may look like when ordering groceries through an app:

Figure 2.5 – Example request through a simplified ordering system

Each trace represents a unique request through a system that can be either synchronous or asynchronous. Synchronous requests occur in sequence with each unit of work completed before continuing. An example of a synchronous request may be of a client application making a call to a server and waiting or blocking until a response is returned before...

Metrics

Just as distributed traces do, metrics provide information about the state of a running system to developers and operators. The data collected via metrics can be aggregated over time to identify trends and patterns in applications graphed through various tools and visualizations. The term metrics has a broad range of applications as they can capture low-level system metrics such as CPU cycles, or higher-level details such as the number of blue sweaters sold today. These examples would be helpful to different groups in an organization.

Additionally, metrics are critical to monitoring the health of an application and deciding when an on-call engineer should be alerted. They form the basis of service level indicators (SLIs) (https://en.wikipedia.org/wiki/Service_level_indicator) that measure the performance of an application. These indicators are then used to set service level objectives (SLOs) (https://en.wikipedia.org/wiki/Service-level_objective) that organizations use to...

Logs

Although logs have evolved, what constitutes a log is quite broad. Also known as log files, a log is a record of events written to output. Traditionally, logs would be written to a file on disk, searching through as needed. A more recent practice is to emit logs to remote services using the network. This provides long-term storage for the data in a location and improves searchability and aggregation.

Anatomy of a log

Many applications define their formats for what constitutes a log. There are several existing standard formats. An example includes the Common Log Format often used by web servers. It's challenging to identify commonalities across formats, but at the very least, a log should consist of the following:

• A timestamp recording the time of the event
• The message or payload representing the event

This message can take many forms and include various application-specific information. In the case of structured logging, the log is formatted as a...

Summary

This chapter allowed us to learn or review some concepts that will assist us when instrumenting applications using OpenTelemetry. We looked at the building blocks of distributed tracing, which will come in handy when we go through instrumenting our first application with OpenTelemetry in Chapter 4, Distributed Tracing – Tracing Code Execution. We also started analyzing tracing data using tools that developers and operators make use of every day.

We then switched to the metrics signal; first, looking at the minimal contents of a metric, then comparing different data types commonly used to produce metrics and their structures. Discussing exemplars gave us a brief introduction to how correlating metrics with traces can create a more complete picture of what is happening within a system by combining telemetry across signals.

Looking at log formats and searching through logs to find information about the demo application allowed us to get familiar with yet another tool...