Chapter 10. Clustering and Classifying Text

In the last chapter we studied topic models and how they can help us in organizing and better understanding our documents and its sub-structure. We will now move on to our next set of machine learning algorithms, and for two particular tasks — clustering and classification. We will learn what the intuitive reasoning of these two tasks is, as well as how to perform these tasks using the popular Python machine learning library, scikit-learn:

• Clustering text
• Classifying text

Clustering and Classifying Text

In the last chapter we studied topic models and how they can help us in organizing and better understanding our documents and its substructure. We will now move on to our next set of machine learning algorithms, and for two particular tasks — clustering and classification. We will learn what the intuitive reasoning of these two tasks is, as well as how to perform these tasks using the popular Python machine learning library, scikit-learn:

• Clustering text
• Classifying text

Clustering text

So far we looked at analyzing text to understand better what the text or corpus consists of. When we tried to POS-tag or NER-tag, we were interested in knowing what kind of words were presented in our documents, and when we topic-modeled, we wanted to know the underlying topics which could be hidden in our texts. Sure, we could use our topic models to attempt to cluster articles, but that isn't its purpose; we would be silly to expect great results if we tried this, too. Remember that since the purpose of topic modeling is to find hidden themes in a corpus and not to group documents together, our methods are not optimized for the task. For example, after we perform topic modeling, a document can be made of 30% topic 1, 30% topic 2, and 40% topic 3. In such a case, we cannot use this information to cluster.

Let us now start exploring how to use machine learning...

Starting clustering

Like every other text analysis algorithm we applied before, the most important step remains the preprocessing step — getting rid of our stop words and lemmatizing words.

Once we're done with this, the next step is to convert our document into a vector representation we are most comfortable with.

Since we're dealing with scikit-learn's implementations for clustering and classification, let us use scikit-learn for our preprocessing. We should also use this opportunity to decide which dataset we intend to use for our experiments. While there are lots of solid options, we will stick with the popular 20 Newsgroups [3] dataset. Since the dataset comes bundled with scikit-learn, loading it and using it becomes an easy task as well.

You can follow the Jupyter notebook [4] on clustering and classification for the full details; we will be using...

K-means

K-means [6] is a classical machine learning algorithm for clustering. It is intuitively easy to understand. Based on a predetermined number of clusters the user decides, it attempts to create clusters. This is done by reducing the distance of points from the respective centroid the point is assigned to. It is an iterative algorithm and keeps doing the process until the centroids and points assigned don't change. It is worth one's time to go through the theory behind the algorithm, though it isn't necessary for us to proceed.

Using K-means with scikit-learn is very easy, and scikit-learn offers two implementations [7] which we can use – either in mini-batches or without. In our code, we allow the user to toggle between which option to use:

minibatch = True
  
if minibatch: 
    km = MiniBatchKMeans(n_clusters=true_k, init='k-means++', n_init...

Hierarchical clustering

Before we dive into hierarchical clustering, it would be a very handy exercise to go through the scikit-learn documentation on clustering [8]. We have to remember that using a different model in scikit-learn is very easy, and that almost all the other steps in the process of clustering remain the same throughout.

We will use Ward's algorithm/method [9] to attempt hierarchical clustering. The algorithm is based on the idea of reducing the variance within each cluster and uses distance measures to do this. Ward's method is one of the earliest methods used in various hierarchical clustering algorithms, which are based on building clusters and arranging them in a hierarchy. In our examples, we will use dendrograms [10] to represent our hierarchical clusters.

To set up our dataset for this method we must first create a matrix with pair-wise distances...

Classifying text

In our previous section, we discussed cluster, which was an unsupervised learning algorithm. Classification, on the other hand, is a supervised learning algorithm. What does supervised and unsupervised mean? In our previous example, we had the labels or the truth values. This is information about which class or label a document actually belongs to. But you would have also noticed we never used this information. When we trained our model, we never used the labels. This kind of learning is called unsupervised learning, and clustering is a popular example of an unsupervised learning task.

In classification problems, we are aware of the classes which we want to assign documents or data points to, and we use this information to train our model. In fact, as we are going to see very soon - there is hardly any change in our approach to clustering and classification, apart...

Summary

And that sums it up! You can now build basic classifiers yourself - the classic problem of classifying emails as spam and not-spam is now something you can replicate yourself. We have seen various clustering algorithms such as K-means, and hierarchal clustering algorithms. We discussed what supervised and unsupervised learning algorithms are, and saw examples of how to run both using scikit-learn.

You can also explore your text data in all sorts of ways with the clustering and topic modeling tools we have. Let's attempt to go one step further in the next chapter - and build a basic information retrieval machine which can search for similar documents.