Introduction to Vectors and Embeddings

In this first chapter, we will dive into the fascinating world of embeddings, or vectors, and their diverse applications across various domains. We’ll introduce the concept of embeddings, which help represent complex data and enable powerful machine learning (ML) models to analyze and process that data. You’ll learn about the roles of supervised and unsupervised learning in creating embeddings and the challenges addressed by vectors. Moreover, we’ll discuss examples illustrating the broad applications of vector representation in different fields. We’ll also introduce you to the ecosystem of tools and platforms that enhance the developer experience when working with vector search, including Hugging Face and various backend considerations.

As we delve deeper into this chapter, you’ll discover the rapidly evolving market landscape and platforms that facilitate the implementation of vector search. We’ll...

Exploring the roles of supervised and unsupervised learning in vector search

Supervised learning is a method in ML where a model learns from labeled data to determine a mapping between input features and output labels. During the training process, the model adjusts its parameters to minimize the error between its predictions and the true labels. Supervised learning is widely used in various applications, such as image classification, speech recognition, and natural language processing (NLP). One of the key challenges with supervised learning is that a curated labeled dataset must be provided to train the model.

Unsupervised learning, on the other hand, involves discovering patterns and structures in input data without using labeled examples. This type of learning focuses on finding similarities, differences, or underlying structures within the data by using techniques such as clustering, dimensionality reduction, and density estimation. Unsupervised learning is commonly applied...

Use cases and domains of application

AI-based search is often overwhelming for a majority of users to get started with, mainly, we believe, because of the lack of standardization – there are a lot of options to get started. In this book, we believe that it would be useful for practitioners to understand where to start by targeting the more mature and adopted techniques, use cases, and domains of application.

We will directly address the scope of possibilities for search, navigating around the jargon carried by the complex space of ML, NLP, and deep learning.

In addition, there is something to keep in mind before jumping into such a project: maintaining a balance between complexity, effort, and cost. By knowing how rapidly the field evolves with new research techniques, the initial investment can be short-lived. In this section, we are going to look at what AI-based search is and also explore the different techniques, such as named entity recognition (NER), sentiment analysis...

How is Elastic playing a role in this space?

Now, what role does Elastic play here? There are multiple ways you can leverage Elastic – including Elasticsearch. Elasticsearch is a distributed and highly scalable data store. Its specialty is information retrieval. It’s one thing to run the preceding code in a notebook – it’s another to make it operational at scale and to be accessed by hundreds, thousands, or millions of users.

Furthermore, training can be long as it requires the algorithm to be able to access a large amount of data, indexed for fast access, at scale. There are very few data stores that are versatile enough to be able to cope with structured and unstructured data at the same time, manage a wide range of data types, and be scalable for ingestion and search.

Elasticsearch is a unique choice, not only because of its technical attributes but also because of its vibrant community and large adoption. As mentioned earlier, the field of AI-based...

Summary

This book is going to guide you through pragmatic, practical examples that you will be able to replicate for your needs, ranging from building a vector-powered search application to applying vectors to domains such as observability and security. It targets a large spectrum of practitioners but is more focused on the operations teams, which deal with specific challenges and could make good use of combining vectors and Elastic.

In this chapter, you learned about the key aspects of NLP, vector search, AI-based search, and its relative techniques. You were also shown pieces of code that you can run and adapt for your business applications. Lastly, you gained a high-level understanding of Elasticsearch’s role in this context as well as how to apply vector search beyond the search use case, such as in terms of observability.

In the next chapter, we will get started by looking at vectors in Elasticsearch. We will explore the current search methodology and see how vectors...