Chapter 9. Unsupervised Learning

This chapter will cover how we can do unsupervised learning using MLlib, Spark's machine learning library.

This chapter is divided into the following recipes:

• Clustering using k-means
• Dimensionality reduction with principal component analysis
• Dimensionality reduction with singular value decomposition

Introduction

The following is Wikipedia's definition of unsupervised learning:

"In machine learning, the problem of unsupervised learning is that of trying to find hidden structure in unlabeled data."

In contrast to supervised learning, where we have labeled data to train an algorithm, in unsupervised learning, we ask the algorithm to find a structure on its own. Let's take a look at the following sample dataset:

As you can see in the preceding graph, the data points are forming two clusters, as follows:

In fact, clustering is the most common type of unsupervised learning algorithm.

Clustering using k-means

Cluster analysis or clustering is the process of grouping data into multiple groups so that the data in one group would be similar to the data in other groups.

The following are a few examples where clustering is used:

• Market segmentation: Dividing the target market into multiple segments so that the needs of each segment can be served better
• Social network analysis: Finding a coherent group of people in the social network for ad targeting through a social networking site, such as Facebook
• Data center computing clusters: Putting a set of computers together to improve performance
• Astronomical data analysis: Understanding astronomical data and events, such as galaxy formations
• Real estate: Identifying neighborhoods based on similar features
• Text analysis: Dividing text documents, such as novels or essays, into genres

The k-means algorithm is best illustrated using imagery, so let's look at our sample figure again:

The first step in k-means is to randomly select two points called...

Dimensionality reduction with principal component analysis

Dimensionality reduction is the process of reducing the number of dimensions or features. A lot of real data contains a very high number of features. It is not uncommon to have thousands of features. So we need to drill down to features that matter.

Dimensionality reduction serves several purposes, such as:

• Data compression
• Visualization

When the number of dimensions is reduced, it reduces the disk and memory footprint. Last but not least, it helps algorithms to run faster. It also helps reduce highly correlated dimensions to one.

Humans can only visualize three dimensions, but data has access to a much higher number of dimensions. Visualization can help find hidden patterns in a particular piece of data. Dimensionality reduction helps visualization by compacting multiple features into one.

The most popular algorithm for dimensionality reduction is principal component analysis (PCA).

Let's look at the following dataset:

Let's say the goal...

Dimensionality reduction with singular value decomposition

Often, the original dimensions do not represent data in the best way possible. As we saw in PCA, you can, sometimes, project data to fewer dimensions and still retain most of the useful information.

Sometimes, the best approach is to align dimensions along the features that exhibit the most number of variations. This approach helps eliminate dimensions that are not representative of the data.

Let's look at the following figure again, which shows the best-fitting line on two dimensions:

The projection line shows the best approximation of the original data with one dimension. If we take the points where the gray line is intersecting with the black line and isolating it, we will have a reduced representation of the original data with as much variation retained as possible, as shown in the following figure:

 

Let's draw a line perpendicular to the first projection line, as shown in the following figure:

This line captures as much variation...