Introducing Foundations of Vector Representation

Vectors and vector representation are at the very core of neural search since the quality of vectors determines the quality of search results. In this chapter, you will learn about the concept of vectors within machine learning (ML). You will see common search algorithms using vector representation as well as their weaknesses and strengths.

We’re going to cover the following main topics in this chapter:

• Introducing vectors in ML
• Measuring the similarity between two vectors
• Local and distributed representations

By the end of this chapter, you will have a solid understanding of how every type of data can be represented in vectors and why this concept is at the very core of neural search.

Technical requirements

This chapter has the following technical requirements:

• A laptop with a minimum of 4 GB RAM (8 GB or more is preferred)
• Python installed with version 3.7, 3.8, or 3.9 on a Unix-like operating system, such as macOS or Ubuntu

The code for this chapter can be found at https://github.com/PacktPublishing/Neural-Search-From-Prototype-to-Production-with-Jina/tree/main/src/Chapter02.

Introducing vectors in ML

Text is an important means of recording human knowledge. As of June 2021, the number of web pages indexed by mainstream search engines such as Google and Bing has reached 2.4 billion, and the majority of information is stored as text. How to store this textual information, and even how to efficiently retrieve the required information from the repository, has become a major issue in information retrieval. The first step in solving these problems lies in representing text in a format that is comprehensible to computers.

As network-based information has become increasingly diverse, in addition to text, web pages contain a large amount of multimedia information, such as pictures, music, and video files. These files are more diverse than text in terms of form and content and satisfy users’ needs from different perspectives. How to represent and retrieve these types of information, as well as how to pinpoint the multimodal information needed by users from...

Measuring similarity between two vectors

Measuring similarity between two vectors is important in a neural search system. Once all of the documents have been indexed into their vector representation, given a user query, we carry out the same encoding process to the query. In the end, we compare the encoded query vector against all the encoded document vectors to find out what the most similar documents are.

We can continue our example from the previous section, trying to measure the similarity between doc1 and doc2. First of all, we need to run the script two times to encode both doc1 and doc2:

doc1 = 'Jina is a neural search framework'
doc2 = 'Jina is built with cutting age technology called deep learning'

Then, we can produce a vector representation for both of them:

encoded_doc1 = [1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0]
encoded_doc2 = [1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1]

Since the dimension of the encoded result is always identical...

Local and distributed representations

In this section, we’ll dive into local representations and distributed representations. We will go through the characteristics of two different representations and list the most widely used local and global representations to encode different modalities of data.

Local vector representation

As a classic method of text representation, local representation only makes use of the disjointed dimensions in the vector for a certain word when it is represented as a vector. Disjointed dimension means that each dimensionality of the vector represents a single token.

When only one dimension is used, it is called one-hot representation. One-hot means that the word is represented as a long vector, and the dimension of the vector is the total number of words to be represented. Most dimensions are 0, while only one dimension has a value of 1. Different words with a dimension of 1 are not used. If this method of representation is stored sparsely...

Summary

This chapter described the method of vector representation, which is a major step in the operation of search engines.

First, we introduced the importance of vector representation and how to use it, and then addressed local and distributed vector representation algorithms. In terms of distributed vector representation, the commonly used representation algorithms for text, images, and audio were covered, and common representation methods for other modalities and multimodality were summarized. Hence, we found that the dense vector representation method often entails relatively rich contextual information when compared with sparse vectors.

When building a scalable neural search system, it is important to create an encoder that can encode raw documents into high-quality embeddings. This encoding process needs to be performed fast to reduce the indexing time. At search time, it is critical to apply the same encoding process and find the top-ranked documents in a reasonable...

Further reading

• Devlin, Jacob, et al. “Bert: Pre-training of deep bidirectional transformers for language understanding.” arXiv preprint arXiv:1810.04805 (2018).
• Simonyan, Karen and Andrew Zisserman. “Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv:1409.1556 (2014).
• He, Kaiming et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
• Schneider, Steffen, et al. “wav2vec: Unsupervised pre-training for speech recognition.” arXiv preprint arXiv:1904.05862 (2019).
• Radford, Alec et al. “Learning transferable visual models from natural language supervision.” International Conference on Machine Learning. PMLR, 2021.