Fundamentals of text embedding with NLKT and Gensim
In this section, we will go through the fundamentals of text embedding: tokenizing a book, embedding the tokens, and exploring the vector space we created.
Open Embedding_with_NLKT_Gensim.ipynb in the chapter directory of the GitHub repository.
We will first install the libraries we will need.
Installing libraries
The program first installs the Natural Language Toolkit (NLTK):
!pip install --upgrade nltk -qq
import nltk
The NLTK will take us down to the token level as in Chapter 10, Investigating the Role of Tokenizers in Shaping Transformer Models.
We’ll use the punkt sentence tokenizer:
nltk.download('punkt')
The program installs gensim for the similarity tools:
!pip install gensim -qq
import gensim
print(gensim.__version__)
The output is the version:
4.3.2
The first step is to read the file.
