Multiclass Logistic Regression

The extension to Logistic Regression, for classifying more than two classes, is Multiclass Logistic Regression. Its foundation is actually a generic approach: it doesn't just work for Logistic Regressors, it also works with other binary classifiers. The base algorithm is named One-vs-rest, or One-vs-all, and it's simple to grasp and apply.

Let's describe it with an example: we have to classify three kinds of flowers and, given some features, the possible outputs are three classes: f1, f2, and f3. That's not what we've seen so far; in fact, this is not a binary classification problem. Instead, it seems very easy to break down this problem into three simpler problems:

• Problem #1: Positive examples (that is, the ones that get the label "1") are f1; negative examples are all the others

• Problem #2: Positive examples are f2; negative examples are f1 and f3

• Problem #3: Positive examples are f3; negative examples are f1 and f2

For all three problems, we can use a binary...