Exploring Neural Networks with Activation Atlases
For example, Karpathy’s CNN codes visualization gives a global view of a dataset by taking each image and organizing them by their activation values from a neural network. Roughly speaking, we can think of feature visualization as creating an idealized image of what the network thinks would produce a particular activation vector. Whereas we normally use a network to transform an image into an activation vector, in feature visualization we go in the opposite direction.
Source: distill.pub