Chapter 4. Gensim – Vectorizing Text and Transformations and n-grams

• Introducing Gensim
• Vectors and why we need them
• Vector transformations in Gensim
• n-grams and some more preprocessing
• Summary

References

[1] Vectors:
https://en.wikipedia.org/wiki/Euclidean_vector

[2] Scalability of Semantic Analysis in Natural Language Processing:
https://radimrehurek.com/phd_rehurek.pdf

[3] Latent Dirichlet allocation:
https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation

[4] Latent semantic indexing:
https://en.wikipedia.org/wiki/Latent_semantic_analysis#Latent_semantic_indexing

[5] Random Projection:
https://en.wikipedia.org/wiki/Random_projection

[6] Stanford TMT:
https://nlp.stanford.edu/software/tmt/tmt-0.4/

[7] Gensim notebooks:
https://github.com/RaRe-Technologies/gensim/tree/develop/docs/notebooks

[8] Jupyter Notebooks:
http://jupyter-notebook.readthedocs.io/en/stable/notebook.html

[9] Vector Space Models:
https://en.wikipedia.org/wiki/Vector_space_model

[10] Bayesian Probability:
https://en.wikipedia.org/wiki/Bayesian_probability

[11] TF-IDF:
https://en.wikipedia...

Vectors and why we need them

We're now moving toward the machine learning part of text analysis - this means that we will now start playing a little less with words and a little more with numbers. Even when we used spaCy, the POS-tagging and NER-tagging, for example, was done through statistical models - but the inner workings were largely hidden for us - we passed over Unicode text and after some magic, we have annotated text.

For Gensim however, we're expected to pass vectors as inputs to the IR algorithms (such as LDA or LSI), largely because what's going on under the hood is mathematical operations involving matrices. This means that we have to represent what was previously a string as a vector - and these kind of representations or models are called Vector Space Models [9].

From a mathematical perspective, a vector is a geometric object that has magnitude and direction. We don't need to pay as much attention to this, and rather think of vectors as a way of projecting words onto a mathematical...

Vector transformations in Gensim

Now that we know what vector transformations are, let's get used to creating them, and using them. We will be performing these transformations with Gensim, but even scikit-learn can be used. We'll also have a look at scikit-learn's approach later on.

Let's create our corpus now. We discussed earlier that a corpus is a collection of documents. In our examples, each document would just be one sentence, but this is obviously not the case in most real-world examples we will be dealing with. We should also note that once we are done with preprocessing, we get rid of all punctuation marks - as for as our vector representation is concerned, each document is just one sentence.

Of course, before we start, be sure to install Gensim. Like spaCy, pip or conda is the best way to do this based on your working environment.

from gensim import corpora
documents = [u"Football club Arsenal defeat local rivals this weekend.", u"Weekend football frenzy takes over London.", u"Bank...

n-grams and some more preprocessing

When working with textual data, context can be very important. As we discussed before, we sometimes lose this context in vector representations, knowing only the count of each word. N-grams, and in particular, bi-grams are going to help us solve this problem, at least to some extent.

An n-gram is a contiguous sequence ofnitems in the text. In our case, we will be dealing with words being the item, but depending on the use case, it could be even letters, syllables, or sometimes in the case of speech, phonemes. A bi-gram is whenn = 2.

One way bi-grams are calculated in the text is by calculating the conditional probability of a token given by the preceding token. It can also just be calculated by choosing words that appear next to each other, but it is more useful for us to use bi-grams that are more likely to appear as a pair. Such a bi-gram is called a collocation. What this means is that we're trying to find pairs of words that are more likely to appear...

Summary

We've seen in this chapter why it makes sense to change our representation of text from words to numbers, and why this is the only language a computer understands. There are different ways computers can interpret words, and TF-IDF and bag of words are two such vector representations. Gensim is a Python package that offers us ways to generate such vector representations, which are later used as inputs into various machine learning and information retrieval algorithms.

There are further preprocessing techniques such as creating n-grams, collocations and removing low-frequency words, which can help us arrive at better results. The concepts of vectors form a basis in natural language processing and we can now get back to using spaCy's pipelines; indeed, Chapter 5, POS-Tagging and Its Applications, Chapter 6, NER-Tagging and Its Applications, andChapter 7, Dependency Parsing, all showcase the power of spaCy, and we will start with POS-tagging algorithms using spaCy.

References

[1] Vectors:https://en.wikipedia.org/wiki/Euclidean_vector

[2] Scalability of Semantic Analysis in Natural Language Processing: https://radimrehurek.com/phd_rehurek.pdf

[3] Latent Dirichlet allocation: https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation

[4] Latent semantic indexing: https://en.wikipedia.org/wiki/Latent_semantic_analysis#Latent_semantic_indexing

[5] Random Projection: https://en.wikipedia.org/wiki/Random_projection

[6] Stanford TMT: https://nlp.stanford.edu/software/tmt/tmt-0.4/

[7] Gensim notebooks: https://github.com/RaRe-Technologies/gensim/tree/develop/docs/notebooks

[8] Jupyter Notebooks: http://jupyter-notebook.readthedocs.io/en/stable/notebook.html

[9] Vector Space Models: https://en.wikipedia.org/wiki/Vector_space_model

[10] Bayesian Probability: https://en.wikipedia.org/wiki/Bayesian_probability

[11] TF-IDF: https://en.wikipedia.org/wiki/Tf-idf

[12] The Amazing power of word vectors: https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/

[13...