Chapter 3. spaCy's Language Models

While we introduced text analysis in Chapter 1, What is Text Analysis?, we did not discuss any of the technical details behind building a text analysis pipeline. In this chapter, we will introduce you to spaCy's language model – these will serve as the first step in text analysis and are the first building block in our pipelines. In this chapter, we will introduce the reader to spaCy and how we can use spaCy to help us in our text analysis tasks, as well as talk about some of its more powerful functionalities, such as Part of Speech-tagging and Named Entity Recognition-tagging. We will finish up with an example of how we can preprocess data quickly and efficiently using the natural language processing Python library, spaCy.

We will cover the following topics in this chapter:

• spaCy
• Installation
• Tokenizing Text
• Summary
• References

spaCy

Having discussed some of the basics of text analysis, let's dive head first into our first Python package we'll be learning to use - spaCy [1].

spaCy describes itself asIndustrial Strength Natural Language Processing – and it most certainly does its best to live up to this promise. Focused on getting things done rather than a more academic approach, spaCy ships with only one part-of-speech tagging algorithm and only one named-entity-recognizer (per language). What this also means is that the package is not bloated with unnecessary features.

We previously mentionedacademic approach – what does this mean? A large number of the open-source packages in the natural language processing and machine learning are usually created or maintained by researchers and those working in academia. While they do end upworking– the aim of the projects is not to provide state-of-the-art implementations of algorithms.NLTK [2] is one such example, where the primary focus of the library is to give students and...

Installation

Let's get started with setting up and installing spaCy. spaCy is compatible with 64-bit CPython [8] 2.6+∕3.3+ and runs on Unix/Linux, macOS/OS X, and Windows. CPython is a reference implementation of Python written in C – we don't need to know the details behind it, and if you have a stable installation of Python running, it is likely your CPython modules are just fine as well. The latest spaCy releases are available over Pip [9] (source packages only) and Conda [10]. Pip and conda are two Python package distributors. Installation requires a working build environment. We will be using Python 3, though the examples are all valid for Python 2 as well.

Pip remains the most straightforward choice, but for users with anaconda installed, they will be using conda instead.

pip install -U spacy

Note

When using pip, it is generally recommended that you install packages in a virtualenv tool to avoid modifying system state.

Since we will be downloading a number of Python packages throughout...

Tokenizing text

You can see that the first step in this pipeline is tokenizing – what exactly is this?

Tokenization is the task of splitting a text into meaningful segments, called tokens. These segments could be words, punctuation, numbers, or other special characters that are the building blocks of a sentence. In spaCy, the input to the tokenizer is a Unicode text, and the output is aDoc object [19].

Different languages will have different tokenization rules. Let's look at an example of how tokenization might work in English. For the sentence – Let us go to the park., it's quite straightforward, and would be broken up as follows, with the appropriate numerical indices:

0	1	2	3	4	5	6
Let	us	go	to	the	park	.

 

This looks awfully like the result when we just run text.split(' ')– when does tokenizing involve more effort?

If the previous sentence was Let's go to the park. instead, the tokenizer would have to be smart enough to split Let's into Let and 's. This means that there are some special rules to follow...

Summary

spaCy offers us an easy way to annotate your text data very easily, and with the language model, we annotate your text data with a lot of information – not just tokenizing and whether it is a stop word or not, but also the part of speech, named entity tag, and so on – we can also train these annotating models on our own, giving a lot of power to the language model and processing pipeline! Downloading the models and using virtual environments are also an important part of this process. We will now move on to using our cleaned data in a way that machines can understand us – with vectors, and what kind of Python libraries we would need for the same.

References

[1] spaCy: https://spacy.io

[2] NLTK: http://www.nltk.org

[3] Dead Code should be buried: https://explosion.ai/blog/dead-code-should-be-buried

[4] Pattern: https://www.clips.uantwerpen.be/pattern

[5] Wrapper Function: https://en.wikipedia.org/wiki/Wrapper_function

[6] Brown Corpus: https://en.wikipedia.org/wiki/Brown_Corpus

[7] NLTK Corpus: http://www.nltk.org/book/ch02.html

[8] CPython: https://en.wikipedia.org/wiki/CPython

[9] PyPi: https://pypi.python.org/pypi

[10] Conda: https://conda.io/docs/

[11] Virtual Environments:http://docs.python-guide.org/en/latest/dev/virtualenvs/

[12] Can't Pip Install: https://github.com/explosion/spaCy/issues/269

[13] Failed building wheel for spacy: https://stackoverflow.com/questions/43370851/failed-building-wheel-for-spacy

[14] spaCy Facts & Figures: https://alpha.spacy.io/usage/facts-figures

[15] spaCy Language Models: https://alpha.spacy.io/usage/models#languages

[16] spaCy Models overview: https://alpha.spacy.io/models/

[17] spaCy Models releases: https...

The rest of the chapter is locked

Summary

spaCy offers us an easy way to annotate your text data very easily, and with the language model, we annotate your text data with a lot of information – not just tokenizing and whether it is a stop word or not, but also the part of speech, named entity tag, and so on – we can also train these annotating models on our own, giving a lot of power to the language model and processing pipeline! Downloading the models and using virtual environments are also an important part of this process. We will now move on to using our cleaned data in a way that machines can understand us – with vectors, and what kind of Python libraries we would need for the same.

References

[1] spaCy: https://spacy.io

[2] NLTK: http://www.nltk.org

[3] Dead Code should be buried: https://explosion.ai/blog/dead-code-should-be-buried

[4] Pattern: https://www.clips.uantwerpen.be/pattern

[5] Wrapper Function: https://en.wikipedia.org/wiki/Wrapper_function

[6] Brown Corpus: https://en.wikipedia.org/wiki/Brown_Corpus

[7] NLTK Corpus: http://www.nltk.org/book/ch02.html

[8] CPython: https://en.wikipedia.org/wiki/CPython

[9] PyPi: https://pypi.python.org/pypi

[10] Conda: https://conda.io/docs/

[11] Virtual Environments:http://docs.python-guide.org/en/latest/dev/virtualenvs/

[12] Can't Pip Install: https://github.com/explosion/spaCy/issues/269

[13] Failed building wheel for spacy: https://stackoverflow.com/questions/43370851/failed-building-wheel-for-spacy

[14] spaCy Facts & Figures: https://alpha.spacy.io/usage/facts-figures

[15] spaCy Language Models: https://alpha.spacy.io/usage/models#languages

[16] spaCy Models overview: https://alpha.spacy.io/models/

[17] spaCy Models releases: https...