Discriminant Analysis

In the previous chapter, we discussed discrete regression models, including classification using logistic regression. In this chapter, we will begin with an overview of probability, expanding into conditional and independent probability. We then discuss how these two approaches to understanding the laws of probability form the basis for Bayes’ Theorem, which is used directly to expand an approach called Bayesian statistics. Following this topic, we dive into Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA), two powerful classifiers that model data using the Bayesian approach to probability modeling.

In this chapter we’re going to cover the following main topics:

• Bayes’ Theorem
• LDA
• QDA

Bayes’ theorem

In this section, we will discuss Bayes’ Theorem, which is used in the classification models described later in this chapter. We will start the chapter by discussing the basics of probability. Then, we will take a look at dependent events and discuss how Bayes’ Theorem is related to dependent events.

Probability

Probability is a measurement of the likelihood that an event occurs or a certain outcome occurs. Generally, we can group events into two types of events: independent events and dependent events. The distinction between the types of events is in the name. An independent event is an event that is not affected or influenced by the occurrences of other events, while a dependent event is affected or influenced by the occurrences of other events.

Let’s think about some examples of these events. For the first example, think about a fair coin toss. A coin toss can result in one of two states: heads and tails. If the coin is fair, there...

Linear Discriminant Analysis

In the previous chapter, we discussed logistic regression as a classification model leveraging linear regression to model directly the probability of a target distribution given an input distribution. One alternative to this approach is LDA. LDA models the probability of target distribution class memberships given input variable distributions corresponding to each class using decision boundaries constructed using Bayes’ Theorem, which we discussed previously. Where we have k classes, using Bayes’ Theorem, we have the probability density function for LDA class membership simply as P(Y = k|X = x) for any discrete random variable, X. This relies on the posterior probability that an observation x in variable X belongs to the kth class.

Before proceeding, we must first make note that LDA makes three pertinent assumptions:

• Each input variable is normally distributed.
• Across all target classes, there is equal covariance among the predictors...

Quadratic Discriminant Analysis

In the last section, we discussed LDA. The data within each class needs to be drawn from a multivariate Gaussian distribution, and the covariance matrix is the same across different classes. In this section, we consider another type of discriminant analysis called QDA but the assumptions for QDA can be relaxed on the covariance matrix assumption. Here, we do not need the covariance matrix to be identical across different classes but only for each class to have its own covariance matrix. The multivariate Gaussian distribution with a class-specific mean vector within each class for observations is still required to conduct QDA. We assume that an observation from a k th class satisfies the following formula:

X~N(μ k, Σ k)

We’ll thus consider a generative classifier, as follows:

p(X | y = k, θ) = N(X | μ k, Σ k)

And then, its corresponding class posterior is this:

p(y = k | X, ...

Summary

In this chapter, we began with an overview of probability. We covered the differences between conditional and independent probability and how Bayes’ Theorem leverages these concepts to provide a unique approach to probability modeling. Next, we discussed LDA, its assumptions, and how the algorithm can be used to apply Bayesian statistics to both perform classification modeling and supervised dimension reduction. Finally, we covered QDA, an alternative to LDA when linear decision boundaries are not effective.

In the next chapter, we will introduce the fundamentals of time-series analysis, including an overview of the depths and limitations of this approach to answering statistical questions.