Embeddings convert any form of data (text, images, or audio) into real numbers. Thus, a document is converted into a vector. These mathematical representations of documents allow us to calculate the distances between documents and retrieve similar data.

The raw data (books, articles, blogs, pictures, or songs) is first collected and cleaned to remove noise. The prepared data is then fed into a model such as OpenAI text-embedding-3-small, which will embed the data. Activeloop Deep Lake, for example, which we will implement in this chapter, will break a text down into pre-defined chunks defined by a certain number of characters. The size of a chunk could be 1,000 characters, for instance. We can let the system optimize these chunks, as we will implement them in the Optimizing chunking section of the next chapter. These chunks of text make it easier to process large amounts of data and provide more detailed embeddings of a document, as...

A RAG pipeline will typically collect data and prepare it by cleaning it, for example, chunking the documents, embedding them, and storing them in a vector store dataset. The vector dataset is then queried to augment the user input of a generative AI model to produce an output. However, it is highly recommended not to run this sequence of RAG in one single program when it comes to using a vector store. We should at least separate the process into three components:

• Data collection and preparation
• Data embedding and loading into the dataset of a vector store
• Querying the vectorized dataset to augment the input of a generative AI model to produce a response

Let’s go through the main reasons for this component approach:

• Specialization, which will allow each member of a team to do what they are best at, either collecting and cleaning data, running embedding models, managing vector stores, or tweaking generative...

Let’s dive into what a real-life RAG pipeline looks like. Imagine we’re a team that has to deliver a whole system in just a few weeks. Right off the bat, we’re bombarded with questions like:

• Who’s going to gather and clean up all the data?
• Who’s going to handle setting up OpenAI’s embedding model?
• Who’s writing the code to get those embeddings up and running and managing the vector store?
• Who’s going to take care of implementing GPT-4 and managing what it spits out?

Within a few minutes, everyone starts looking pretty worried. The whole thing feels overwhelming—like, seriously, who would even think about tackling all that alone?

So here’s what we do. We split into three groups, each of us taking on different parts of the pipeline, as shown in Figure 2.3:

Figure 2.3: RAG pipeline components

Each of the three groups has one...

We will now build a RAG pipeline by implementing the pipeline described in the previous section and illustrated in Figure 2.3. We will implement three components assuming that three teams (Team #1, Team #2, and Team #3) work in parallel to implement the pipeline:

• Data collection and preparation by Team #1
• Data embedding and storage by Team #2
• Augmented generation by Team #3

The first step is to set up the environment for these components.

Setting up the environment

Let’s face it here and now. Installing cross-platform, cross-library packages with their dependencies can be quite challenging! It is important to take this complexity into account and be prepared to get the environment running correctly. Each package has dependencies that may have conflicting versions. Even if we adapt the versions, an application may not run as expected anymore. So, take your time to install the right versions of the packages and dependencies...

In this section, we will implement cosine similarity to measure the similarity between user input and the generative AI model’s output. We will also measure the augmented user input with the generative AI model’s output. Let’s first define a cosine similarity function:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
def calculate_cosine_similarity(text1, text2):
    vectorizer = TfidfVectorizer()
    tfidf = vectorizer.fit_transform([text1, text2])
    similarity = cosine_similarity(tfidf[0:1], tfidf[1:2])
    return similarity[0][0]

Then, let’s calculate a score that measures the similarity between the user prompt and GPT-4’s response:

similarity_score = calculate_cosine_similarity(user_prompt, gpt4_response)
print(f"Cosine Similarity Score: {similarity_score:.3f}")

The score is low, although the output seemed...

In this chapter, we tackled the complexities of using RAG-driven generative AI, focusing on the essential role of document embeddings when handling large datasets. We saw how to go from raw texts to embeddings and store them in vector stores. Vector stores such as Activeloop, unlike parametric generative AI models, provide API tools and visual interfaces that allow us to see embedded text at any moment.

A RAG pipeline detailed the organizational process of integrating OpenAI embeddings into Activeloop Deep Lake vector stores. The RAG pipeline was broken down into distinct components that can vary from one project to another. This separation allows multiple teams to work simultaneously without dependency, accelerating development and facilitating specialized focus on individual aspects, such as data collection, embedding processing, and query generation for the augmented generation AI process.

We then built a three-component RAG pipeline, beginning by highlighting the...

Answer the following questions with Yes or No:

1. Do embeddings convert text into high-dimensional vectors for faster retrieval in RAG?
2. Are keyword searches more effective than embeddings in retrieving detailed semantic content?
3. Is it recommended to separate RAG pipelines into independent components?
4. Does the RAG pipeline consist of only two main components?
5. Can Activeloop Deep Lake handle both embedding and vector storage?
6. Is the text-embedding-3-small model from OpenAI used to generate embeddings in this chapter?
7. Are data embeddings visible and directly traceable in an RAG-driven system?
8. Can a RAG pipeline run smoothly without splitting into separate components?
9. Is chunking large texts into smaller parts necessary for embedding and storage?
10. Are cosine similarity metrics used to evaluate the relevance of retrieved information?

• OpenAI Ada documentation for embeddings: https://platform.openai.com/docs/guides/embeddings/embedding-models
• OpenAI GPT documentation for content generation: https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4
• Activeloop API documentation: https://docs.deeplake.ai/en/latest/
• MiniLM model reference: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

• OpenAI’s documentation on embeddings: https://platform.openai.com/docs/guides/embeddings
• Activeloop documentation: https://docs.activeloop.ai/

Join our community on Discord

Join our community’s Discord space for discussions with the author and other readers:

https://www.packt.link/rag