Monotonic Constraints and Model Tuning for Interpretability

Most model classes have hyperparameters that can be tuned for faster execution speed, increasing predictive performance, and reducing overfitting. One way of reducing overfitting is by introducing regularization into the model training. In Chapter 3, Interpretation Challenges, we called regularization a remedial interpretability property, which reduces complexity with a penalty or limitation that forces the model to learn sparser representations of the inputs. Regularized models generalize better, which is why it is highly recommended to tune models with regularization to avoid overfitting to the training data. As a side effect, regularized models tend to have fewer features and interactions, making the model easier to interpret—less noise means a clearer signal!

And even though there are many hyperparameters, we will only focus on those that improve interpretability by controlling overfitting. Also, to a certain...

Technical requirements

This chapter’s example uses the mldatasets, pandas, numpy, sklearn, xgboost, lightgbm, catboost, tensorflow, bayes_opt, tensorflow_lattice, matplotlib, seaborn, scipy, xai, and shap libraries. Instructions on how to install these libraries are in the preface.

The code for this chapter is located here: https://packt.link/pKeAh

The mission

The issue of algorithmic fairness is one with massive social implications, from the allocation of welfare resources to the prioritization of life-saving surgeries to screening job applications. These machine learning algorithms can determine a person’s livelihood or life, and it’s often the most marginalized and vulnerable populations that get the worst treatment from these algorithms because they perpetuate systemic biases learned from the data. Therefore, it’s poorer families that get misclassified for child abuse; it’s racial-minority people who get underprioritized for medical treatment; and it’s women who get screened out of high-paying tech jobs. Even in cases involving less immediate and individualized risks such as online searches, Twitter/X bots, and social media profiles, societal prejudices such as elitism, racism, sexism, and ageism are reinforced.

This chapter will continue on the mission from Chapter 6, Anchors and Counterfactual...

The approach

You have decided to take a three-fold approach, as follows:

• Placing guardrails with feature engineering: Leveraging lessons learned in Chapter 6, Anchors and Counterfactual Explanations, as well as the domain knowledge we already have about priors and age, in particular, we will engineer some features.
• Tuning models for interpretability: Once the data is ready, we will tune many models with different class weighting and overfitting prevention techniques. These methods will ensure that the models not only generalize better but are also easier to interpret.
• Implementing model constraints: Last but not least, we will implement monotonic and interaction constraints on the best models to make sure that they don’t stray from trusted and fair interactions.

In the last two sections, we will make sure the models perform accurately and fairly. We will also compare recidivism risk distributions between the data and the model to ensure that...

The preparations

You will find the code for this example here: https://github.com/PacktPublishing/Interpretable-Machine-Learning-with-Python-2E/blob/main/12/Recidivism_part2.ipynb

Loading the libraries

To run this example, you need to install the following libraries:

• mldatasets to load the dataset
• pandas and numpy to manipulate it
• sklearn (scikit-learn), xgboost, lightgbm, catboost, tensorflow, bayes_opt, and tensorflow_lattice to split the data and fit the models
• matplotlib, seaborn, scipy, xai, and shap to visualize the interpretations

You should load all of them first, as follows:

import math
import os
import copy
import mldatasets
import pandas as pd
import numpy as np
from sklearn import preprocessing, model_selection, metrics,\
    linear_model, svm, neural_network, ensemble
import xgboost as xgb
import lightgbm as lgb
import catboost as cb
import tensorflow as tf
from bayes_opt import BayesianOptimization
import tensorflow_lattice...

Placing guardrails with feature engineering

In Chapter 6, Anchors and Counterfactual Explanations, we learned that besides race, the features most prominent in our explanations were age, priors_count, and c_charge_degree. Thankfully, the data is now balanced, so the racial bias attributed to this imbalance is now gone. However, through anchors and counterfactual explanations, we found some troubling inconsistencies. In the case of age and priors_count, these inconsistencies were due to how those features were distributed. We can correct issues with distribution through feature engineering, and, that way, ensure that a model doesn’t learn from uneven distributions. In c_charge_degree's case, being categorical, it lacked a discernible order, and this lack of order created unintuitive explanations.

In this section, we will study ordinalization, discretization, and interaction terms, three ways in which you can place guardrails through feature engineering.

Ordinalization...

Tuning models for interpretability

Traditionally, regularization was only achieved by imposing penalty terms such as L1, L2, or elastic net on the coefficients or weights, which shrink the impact of the least relevant features. As seen in the Embedded methods section of Chapter 10, Feature Selection and Engineering for Interpretability, this form of regularization results in feature selection while also reducing overfitting. And this brings us to another broader definition of regularization, which does not require a penalty term. Often, this comes as imposing a limitation, or a stopping criterion that forces the model to curb its complexity.

In addition to regularization, both in its narrow (penalty-based) and broad sense (overfitting methods), there are other methods that tune a model for interpretability—that is, improve the fairness, accountability, and transparency of a model through adjustments to the training process. For instance, the class imbalance hyperparameters...

Implementing model constraints

We will discuss how to implement constraints first with XGBoost and all popular tree ensembles, for that matter, because the parameters are named the same (see Figure 12.12). Then, we will do so with TensorFlow Lattice. But before we move forward with any of that, let’s remove race from the data, as follows:

X_train_con = X_train.drop(['race'], axis=1).copy()
X_test_con = X_test.drop(['race'], axis=1).copy()

Now, with race out of the picture, the model may still have some bias. However, the feature engineering we performed and the constraints we will place can help align the model against them, given the double standards we found in Chapter 6, Anchors and Counterfactual Explanations. That being said, the resulting model might perform worse against the test data. There are two reasons for this, outlined here:

• Loss of information: Race, especially through interaction with other features, impacted the outcome...

Mission accomplished

It’s often the data that takes the blame for a poor-performing, uninterpretable, or biased model, and that can be true, but many different things can be done in the preparation and model development stages to improve it. To offer an analogy, it’s like baking a cake. You need quality ingredients, yes. But seemingly small differences in the preparation of these ingredients and baking itself—such as the baking temperature, the container used, and time—can make a huge difference. Hell! Even things that are out of your control, such as atmospheric pressure or moisture, can impact baking! Even after it’s all finished, how many different ways can you assess the quality of a cake?

This chapter is about these many details, and, as with baking, they are part exact science and part art form. The concepts discussed in this chapter also have far-reaching consequences, especially regarding how to optimize a problem that doesn’t...

Summary

After reading this chapter, you should now understand how to leverage data engineering to enhance interpretability, regularization to reduce overfitting, and constraints to comply with policies. The primary end goals are to place guardrails and curb the complexity that hinders interpretability.

In the next chapter, we will look at ways to enhance model reliability through adversarial robustness.

Dataset sources

• ProPublica Data Store (2019). COMPAS Recidivism Risk Score Data and Analysis. Originally retrieved from https://www.propublica.org/datastore/dataset/compas-recidivism-risk-score-data-and-analysis

Further reading

• Hastie, T. J., Tibshirani, R. J. and Friedman, J. H. (2001). The elements of statistical learning. Springer-Verlag, New York, USA
• Wang, S. & Gupta, M. (2020). Deontological Ethics By Monotonicity Shape Constraints. AISTATS. https://arxiv.org/abs/2001.11990
• Cotter, A., Gupta, M., Jiang, H., Ilan, E. L., Muller, J., Narayan, T., Wang, S. & Zhu, T. (2019). Shape Constraints for Set Functions. ICML. http://proceedings.mlr.press/v97/cotter19a.html
• Gupta, M. R., Cotter A., Pfeifer, J., Voevodski, K., Canini, K., Mangylov, A., Moczydlowski, W. and van Esbroeck, A. (2016). Monotonic Calibrated Interpolated Look-Up Tables. Journal of Machine Learning Research 17(109):1−47. https://arxiv.org/abs/1505.06378
• Noble, S. (2018). Algorithms of oppression: data discrimination in the age of Google. NYU Press

Learn more on Discord

To join the Discord community for this book – where you can share feedback, ask the author...