Chapter 3. Logistic Regression Versus Random Forest

In this chapter, we will be making a comparison between logistic regression and random forest, with a classification example of German credit data. Logistic regression is a very popularly utilized technique in the credit and risk industry for checking the probability of default problems. Major challenges nowadays being faced by credit and risk departments with regulators are due to the black box nature of machine learning models, which is slowing down the usage of advanced models in this space. However, by drawing comparisons of logistic regression with random forest, some turnarounds could be possible; here we will discuss the variable importance chart and its parallels to the p-value of logistic regression, also we should not forget the major fact that significant variables remain significant in any of the models on a fair ground, though some change in variable significance always exists between any two models.

Maximum likelihood estimation

Logistic regression works on the principle of maximum likelihood estimation; here, we will explain in detail what it is in principle so that we can cover some more fundamentals of logistic regression in the following sections. Maximum likelihood estimation is a method of estimating the parameters of a model given observations, by finding the parameter values that maximize the likelihood of making the observations, this means finding parameters that maximize the probability p of event 1 and (1-p) of non-event 0, as you know:

probability (event + non-event) = 1

Example: Sample (0, 1, 0, 0, 1, 0) is drawn from binomial distribution. What is the maximum likelihood estimate of μ?

Solution: Given the fact that for binomial distribution P(X=1) = μ and P(X=0) = 1- μ where μ is the parameter:

Here, log is applied to both sides of the equation for mathematical convenience; also, maximizing likelihood is the same as the maximizing log of likelihood:

Determining the maximum...

Logistic regression – introduction and advantages

Logistic regression applies maximum likelihood estimation after transforming the dependent variable into a logit variable (natural log of the odds of the dependent variable occurring or not) with respect to independent variables. In this way, logistic regression estimates the probability of a certain event occurring. In the following equation, log of odds changes linearly as a function of explanatory variables:

One can simply ask, why odds, log(odds) and not probability? In fact, this is interviewers favorite question in analytics interviews.

The reason is as follows:

By converting probability to log(odds), we have expanded the range from [0, 1] to [- ∞, +∞ ]. By fitting model on probability we will encounter a restricted range problem, and also by applying log transformation, we cover-up the non-linearity involved and we can just fit with a linear combination of variables.

One more question one ask is what will happen if someone fit the linear...

Random forest

The random forest (RF) is a very powerful technique which is used frequently in the data science field for solving various problems across industries, as well as a silver bullet for winning competitions like Kaggle. We will cover various concepts from the basics to in depth in the next chapter; here we are restricted to the bare necessities. Random forest is an ensemble of decision trees, as we know, logistic regression has very high bias and low variance technique; on the other hand, decision trees have high variance and low bias, which makes decision trees unstable. By averaging decision trees, we will minimize the variance component the of model, which makes approximate nearest to an ideal model.

RF focuses on sampling both observations and variables of training data to develop independent decision trees and take majority voting for classification and averaging for regression problems respectively. In contrast, bagging samples only observations at random and selects all columns...

Variable importance plot

Variable importance plot provides a list of the most significant variables in descending order by a mean decrease in Gini. The top variables contribute more to the model than the bottom ones and also have high predictive power in classifying default and non-default customers.

Surprisingly, grid search does not have variable importance functionality in Python scikit-learn, hence we are using the best parameters from grid search and plotting the variable importance graph with simple random forest scikit-learn function. Whereas, in R programming, we have that provision, hence R code would be compact here:

>>> import matplotlib.pyplot as plt 
>>> rf_fit = RandomForestClassifier(n_estimators=1000, criterion="gini", max_depth=300, min_samples_split=3,min_samples_leaf=1) 
>>> rf_fit.fit(x_train,y_train)    
>>> importances = rf_fit.feature_importances_ 
>>> std = np.std([tree.feature_importances_ for tree in rf_fit.estimators_...

Comparison of logistic regression with random forest

One major issue facing the credit risk industry from regulators is due to the black box nature of machine learning models. This section focuses upon drawing parallels between logistic regression and random forest models to create transparency for random forest, so that it will be less intimidating for regulators while approving implementation of machine learning models. Last but not least, readers will also be educated on the comparison of statistical models with machine learning models.

In the following table, both models explanatory variables have been put in descending order based on the importance of them towards the model contribution. In the logistic regression model, it is the p-value (minimum is a better predictor), and for random forest it is the mean decrease in Gini (maximum is a better predictor). Many of the variables are very much matching in importance like, status_exs_accnt_A14, credit_hist_A34, Installment_rate_in_percentage_of_disposable_income...

Summary

In this chapter, you have learned the working principles of logistic regression and its step-by-step solving methodology by iteratively removing insignificant and multi-collinear variables to find the best fit by constantly checking AIC and concordance values to determine the best model in a statistical way. Subsequently we looked at machine learning model and random forest being applied to calculate the test accuracy. It was found that, by carefully tuning the hyperparameters of random forest using grid search, we were able to uplift the results by 10 percent in terms of test accuracy from 80 percent from logistic regression to 90 percent from random forest.

In the next chapter, we will be covering complete tree based models such as decision trees, random forest, boosted trees, ensemble of models, and so on to further improve accuracy!