Categorizing text

Text categorization or classification deals with labeling documents to certain predefined classes. One of the most common tasks of text classification is labeling e-mail as ham and spam. We will discuss how to implement text classification in Mahout.

The dataset

For the text classification case study, we are going to use the 20 newsgroups dataset. The data is from transcripts of several months of postings made in 20 Usenet newsgroups on 20 different topics. Download the dataset from http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz.

The dataset is divided into train and test sets, and each set has 20 subdirectories. If you look at the training folder, you will see these 20 subdirectories. Each subdirectory will be considered a class label, and all files belonging to the directory will belong to that class. The following screenshot displays the folders in which files of respective classes as present. The folder name is the class label for documents present...