Building Models with Distance Metrics

This chapter will cover the following recipes:

• Using k-means to cluster data
• Optimizing the number of centroids
• Assessing cluster correctness
• Using MiniBatch k-means to handle more data
• Quantizing an image with k-means clustering
• Finding the closest objects in the feature space
• Probabilistic clustering with Gaussian Mixture Models
• Using k-means for outlier detection
• Using KNN for regression

Introduction

In this chapter, we'll cover clustering. Clustering is often grouped with unsupervised techniques. These techniques assume that we do not know the outcome variable. This leads to ambiguity in outcomes and objectives in practice, but nevertheless, clustering can be useful. As we'll see, we can use clustering to localize our estimates in a supervised setting. This is perhaps why clustering is so effective; it can handle a wide range of situations, and often the results are, for the lack of a better term, sane.

We'll walk through a wide variety of applications in this chapter, from image processing to regression and outlier detection. Clustering is related to classification of categories. You have a finite set of blobs or categories. Unlike classification, you do not know the categories in advance. Additionally, clustering can often be viewed through a...

Using k-means to cluster data

In a dataset, we observe sets of points gathered together. With k-means, we will categorize all the points into groups, or clusters.

Getting ready

First, let's walk through some simple clustering; then we'll talk about how k-means works:

import numpy as np
import pandas as pd
 
from sklearn.datasets import make_blobs
blobs, classes = make_blobs(500, centers=3)

Also, since we'll be doing some plotting, import matplotlib as shown:

import matplotlib.pyplot as plt
%matplotlib inline  #Within an ipython notebook

How to do it…

We...

Optimizing the number of centroids

When doing k-means clustering, we really do not know the right number of clusters in advance, so finding this out is an important step. Once we know (or estimate) the number of centroids, the problem will start to look more like a classification one as our knowledge to work with will have increased substantially.

Getting ready

Evaluating the model performance for unsupervised techniques is a challenge. Consequently, sklearn has several methods for evaluating clustering when a ground truth is known, and very few for when it isn't.

We'll start with a single cluster model and evaluate its similarity. This is more for the purpose of mechanics as measuring the similarity of one cluster...

Assessing cluster correctness

We talked a little bit about assessing clusters when the ground truth is not known. However, we have not yet talked about assessing k-means when the cluster is known. In a lot of cases, this isn't knowable; however, if there is outside annotation, we will know the ground truth or at least the proxy sometimes.

Getting ready

So, let's assume a world where we have an outside agent supplying us with the ground truth.

We'll create a simple dataset, evaluate the measures of correctness against the ground truth in several ways, and then discuss them:

from sklearn import datasets
from sklearn import cluster

blobs, ground_truth = datasets.make_blobs(1000, centers=3,cluster_std=1.75)

...

Using MiniBatch k-means to handle more data

K-means is a nice method to use; however, it is not ideal for a lot of data. This is due to the complexity of k-means. This said, we can get approximate solutions with much better algorithmic complexity using MiniBatch k-means.

Getting ready

MiniBatch k-means is a faster implementation of k-means. K-means is computationally very expensive; the problem is NP-hard.

However, using MiniBatch k-means, we can speed up k-means by orders of magnitude. This is achieved by taking many subsamples that are called MiniBatches. Given the convergence properties of subsampling, a close approximation to regular k-means is achieved provided there are good initial conditions.

...

Quantizing an image with k-means clustering

Image processing is an important topic in which clustering has some application. It's worth pointing out that there are several very good image processing libraries in Python. scikit-image is a sister project of scikit-learn. It's worth taking a look at if you want to do anything complicated.

A big point of this chapter is that images are data as well and clustering can be used to try to guess where some objects in an image are. Clustering can be part of an image processing pipeline.

Getting ready

We will have some fun in this recipe. The goal is to use a cluster to blur an image. First, we'll make use of SciPy to read the image. The image is translated in a three...

Finding the closest object in the feature space

Sometimes, the easiest thing to do is to find the distance between two objects. We just need to find some distance metric, compute the pairwise distances, and compare the outcomes with what is expected.

Getting ready

A lower level utility in scikit-learn is sklearn.metrics.pairwise. It contains server functions used to compute distances between vectors in a matrix X or between vectors in X and Y easily. This can be useful for information retrieval. For example, given a set of customers with attributes of X, we might want to take a reference customer and find the closest customers to this customer.

In fact, we might want to rank customers by the notion of similarity measured by...

Probabilistic clustering with Gaussian mixture models

In k-means, we assume that the variance of the clusters is equal. This leads to a subdivision of space that determines how the clusters are assigned; but what about a situation where the variances are not equal and each cluster point has some probabilistic association with it?

Getting ready

There's a more probabilistic way of looking at k-means clustering. Hard k-means clustering is the same as applying a Gaussian mixture model with a covariance matrix, S, which can be factored to the error times of the identity matrix. This is the same covariance structure for each cluster. It leads to spherical clusters. However, if we allow S to vary, a GMM can be estimated and...

Using k-means for outlier detection

In this recipe, we'll look at both the debate and mechanics of k-means for outlier detection. It can be useful to isolate some types of errors, but care should be taken when using it.

Getting ready

We'll use k-means to do outlier detection on a cluster of points. It's important to note that there are many camps when it comes to outliers and outlier detection. On one hand, we're potentially removing points that were generated by the data-generating process by removing outliers. On the other hand, outliers can be due to a measurement error or some other outside factor.

This is the most credence we'll give to the debate. The rest of this recipe is about finding outliers...

Using KNN for regression

Regression is covered elsewhere in the book, but we might also want to run a regression on pockets of the feature space. We can think that our dataset is subject to several data processes. If this is true, only training on similar data points is a good idea.

Getting ready

Our old friend, regression, can be used in the context of clustering. Regression is obviously a supervised technique, so we'll use K-Nearest Neighbors (KNN) clustering rather than k-means. For KNN regression, we'll use the K closest points in the feature space to build the regression rather than using the entire space as in regular regression.

...