Interpreting Neural Networks

When trying to comprehend the reasons behind a model’s prediction, local per-sample feature importance can be a valuable tool. This method enables you to focus your analysis on a smaller part of the input data, resulting in a more targeted understanding of key features that contributed to the model’s output. However, it is often still unclear which patterns the models are using to identify highly important features. This issue can be somewhat circumvented by reviewing more prediction explanations from targeted samples meant to strategically discern the actual reason for the prediction, which will also be introduced practically later in this chapter. However, this method is limited to the available number of samples you must validate your model against, and it can sometimes still be difficult to pinpoint the pattern used concretely.

Deep neural networks (DNNs) learn low- to high-level features that help the prediction layer discern the right...

Technical requirements

This chapter includes practical implementation in the Python programming language. To complete it, you will need to have a computer with the following libraries installed:

• torchvision
• torch
• torch-lucent==0.1.8
• matplotlib==3.3.0
• captum
• pillow
• numpy

The code files are present on GitHub: https://github.com/PacktPublishing/The-Deep-Learning-Architect-Handbook/tree/main/CHAPTER_12.

Interpreting neurons

Neurons in NN layers produce features that will be consumed by subsequent layers. The features or activations produced are simply an indicator of how prominent a learned pattern is in the input data. But have you ever wondered what the patterns are? Decoding the actual patterns learned by the NN can further improve the transparency needed to achieve the goals mentioned in the Exploring the value of prediction explanations section of Chapter 11, Explaining Neural Network Predictions.

Data is composed of many complicated patterns combined into a single sample. Traditionally, to discern what a neuron is detecting, much input data has to be evaluated and compared against other data so that a qualitative conclusion can be made by humans, which is both time-consuming and hard to get right. This method allows us to pinpoint the actual pattern that causes a high activation value visually, without the disturbance of other highly correlated patterns.

More formally...

Finding neurons to interpret

With millions and billions of neurons in today’s SoTA architectures, it’s impossible to interpret every single neuron, and, frankly, a waste of time. The choice of the neuron to explain should depend on your goal. The following list shows some of the different goals and associated methods for choosing suitable neurons:

• Finding out what a certain prediction label or class pattern looks like: In this case, you should simply choose a neuron specific to the prediction of the target label or class. This is usually done to understand whether the model captured the desired patterns of the class well, or whether it learned irrelevant features. This can also be useful in multilabel scenarios where multiple labels always only exist together, and you want to decouple the labels to understand the input patterns associated with a single label better.
• Wanting to understand the latent reasons why a specific label can be predicted in your dataset...

Interpreting learned image patterns

Interpreting NNs that take in image data enables a new paradigm in interpretation, which is the capability to visualize exactly what a neuron is detecting. In the case of audio input data, interpreting NNs would allow us to audibly represent what a neuron is detecting, similar to how we visualize patterns in image data! Choose neurons you want to understand based on your goal and visualize the patterns it is detecting through iterative optimizing on image data to activate highly for that neuron.

Practically, however, optimizing image data based on a neuron has an issue where the resulting image often produces high-frequency patterns that are perceived to be noisy, uninterpretable, and unaesthetic. High-frequency patterns are defined to be pixels that are high in intensity and change quickly from one to the next. This is largely due to the mostly unconstrained range of values that a pixel can be represented by, and pixels in isolation are not the...

Discovering the counterfactual explanation strategy

Counterfactual explanation or reasoning is a method of understanding and explaining anything in general by considering alternative and counterfactual scenarios or “what-if” situations. In the context of prediction explanations, it involves identifying changes in the input data that would lead to a different outcome. Ideally, the minimal changes should be identified. In the context of NN interpretation, it involves visualizing the opposite of the target label or intermediate latent features. This approach makes sense to use because it closely aligns with how humans naturally explain events and assess causality, which ultimately allows us to comprehend the underlying decision-making process of the model better.

Humans tend to think in terms of cause and effect, and we often explore alternative possibilities to make sense of events or decisions. For example, when trying to understand why a certain decision was made,...

Summary

NN interpretation is a form of a model understanding process that is different from explaining the predictions made by a model. Both manual discovery of real images and optimizing synthetic images to activate highly for the chosen neuron to interpret are techniques that can be applied together to understand the NN. Practically, the interpretation of NNs will be useful when you have goals to reveal the appearance of a particular prediction label or class pattern, gain insight into the factors contributing to the prediction of a specific label in your dataset or all labels in general, and gain a detailed breakdown of the reasons behind a prediction.

There might be hiccups when trying to apply the technique in your use case, so don’t be afraid to experiment with the parameters and components introduced in this chapter in your goal to interpret your NN.

We will explore a different facet of insights that you can obtain from your data and your model in the next chapter...