Next Generation of Observability Powered by Vectors

As we move ahead in our journey to apply vectors to different use cases, we are going to see how to combine them with observability in this chapter. This domain is a part of the solution provided by the Elastic platform. Observability encompasses notions, such as log analytics, metric analytics, and application performance monitoring (APM), with the common purposes of building awareness about a system’s health and helping operation teams to diagnose and remediate the root causes of incidents whenever they occur.

As systems become more complex, the management of these systems equally becomes complex for the operations teams. The data volumes generated by these systems keep increasing and become hard if not impossible for humans to follow, and in addition, are more and more volatile and distributed. One major aspect of any incident management workflow is interaction with data, whether through a query, a dashboard, or an alert...

Introduction to observability and its importance in modern software systems

Before getting into applying vectors to observability, it is important to take the time to understand the high-level concepts of observability itself.

Observability is the ability to understand the internal state of any layer that composes a system based on its external outputs. This was a relatively easy task when these systems were monoliths with multiple layers but produced a centralized log output that described the behavior of this “box.” However, the process became more and more complicated as companies adopted decentralized cloud-native architectures such as microservices. The output for observability here becomes decorrelated, distributed, and composed of multiple signals, logs, metrics, and traces, to list but a few. Most vendors out there, including Elastic, have responded to this by striving to provide an integrated observability experience with the main goal of reducing mean time...

A new approach—applying vectors and embeddings to log analytics

As you learned in Chapter 1, Introduction to Vectors and Embeddings, vectors and embeddings can be used to describe all sorts of notions and concepts, and we can build vector spaces that represent a given domain. When it comes to vectorizing logs, a couple of approaches are available. In the following sections, we will review two approaches that we believe are the most common ones: training or fine-tuning an existing model for logs and generating human-understandable descriptions and vectorizing these descriptions.

Approach 1—training or fine-tuning an existing model for logs

In this approach, an existing language model, such as word2vec, BERT, or GPT, is trained or fine-tuned specifically for log data. This means using a dataset of raw logs to adapt the model’s parameters so that it can effectively capture the unique characteristics and patterns found in log data. The trained or fine-tuned model...

Log vectorization

Log vectorization is the process of transforming logs into embeddings. This process requires a couple of steps, such as generating logs for the test and expanding and using a general model to generate vectors.

In addition, we made the arbitrary choice to do everything in Python here, which gives you the ability to re-execute the same examples in a Google Colab notebook for educational purposes.

All the code from this chapter is available in the chapter 7 folder of this book’s GitHub repository: https://github.com/PacktPublishing/Vector-Search-for-Practitioners-with-Elastic/tree/main/chapter7.

Note that instead of applying the first approach and trying to generate vectors directly from the logs, we will adopt the strategy of expanding them to a human-readable description first, allowing us to avoid the intensive process of model training.

We are now going to learn how to generate synthetic logs.

Synthetic log

With synthetic logs, we enable...

Semantic search on our logs

In this section, we will focus on transforming the expanded logs into vectors in Elasticsearch and then implementing a semantic search functionality on top of the vectorized content. We do this because—remember—our logs are now stored in human-readable language, so we can apply to them the principles of NLP and semantic search we saw earlier.

Building a query using log vectorization

The following code takes the sequence of expanded logs to build a bulk indexing query for Elasticsearch:

# Generate the sequence of JSON documents for a bulk index operation
bulk_index_body = []
for index, log in enumerate(batchCompletion):
document = {
	"_index": "logs",
	"pipeline": "vectorize-log",
	"_source": {
		"text_field": log, "log": logs[index]
	}
}
bulk_index_body.append(document)

The code then executes the bulk indexing operation using a Python helper. Note that we do not...

Summary

In this chapter, you hopefully have learned the end-to-end process of setting up log vectorization for your log analysis workload based on Elastic. One important point for you to decide on while doing this is whether you expand the log on write or on read, meaning preparing the data while ingesting it or expanding to the meaning of the raw log only when querying it. As you can appreciate, this is still an exploratory domain where applying vector search, or GenAI, to accelerate observability incident management workflow is just beginning to murmur. But you are now prepared with the necessary guidance to implement it as it grows.

In the next chapter, we will address another domain of application for vectors and semantic search—cybersecurity, where the requirements are pretty similar to observability in terms of data, but the workflow is quite different.