Understanding observability

Observability is a big word. What does it mean in practice? There are different definitions out there and big debates regarding how monitoring and observability are similar and different. I take the stance that observability is the property of the system that defines what we can tell about the state and behavior of the system, right now and historically. In particular, we are interested in the health of the system and its components. Monitoring is the collection of tolls, processes, and techniques we use to increase the observability of the system.

There are different facets of information that we need to collect, record, and aggregate in order to get a good sense of what our system is doing. Those facets include logs, metrics, distributed traces, and errors. The monitoring or observability data is multi-dimensional and crosses many levels. Just collecting it doesn't help much. We need to be able to query it, visualize it, and alert other systems...