Interpreting neurons
Neurons in NN layers produce features that will be consumed by subsequent layers. The features or activations produced are simply an indicator of how prominent a learned pattern is in the input data. But have you ever wondered what the patterns are? Decoding the actual patterns learned by the NN can further improve the transparency needed to achieve the goals mentioned in the Exploring the value of prediction explanations section of Chapter 11, Explaining Neural Network Predictions.
Data is composed of many complicated patterns combined into a single sample. Traditionally, to discern what a neuron is detecting, much input data has to be evaluated and compared against other data so that a qualitative conclusion can be made by humans, which is both time-consuming and hard to get right. This method allows us to pinpoint the actual pattern that causes a high activation value visually, without the disturbance of other highly correlated patterns.
More formally...
