The following image shows the top nine activation maps per grid, associated with specific inputs, for the second layer of a ConvNet. On the left, you can think of the mini-grids as activations of individual neurons, for given inputs. The corresponding colored grids on the right relate to the inputs these neurons were shown. What we are visualizing here is the kind of input that maximizes the activation of these neurons. We notice that already some pretty-clear circle detector neurons are visible (grid 2, 2), being activated for inputs such as the top of lamp shades and animal eyes:
Similarly, we notice some square-like pattern detectors (grid 4, 4) that seem to activate for images containing door and window frames. As we progressively visualize activation maps for deeper layers in CNNs, we observe even more complex geometric patterns being picked up...
