Generative Modeling with Sparse Transformers

While it is believed that this allows Transformers to be more flexible than models with fixed connectivity patterns, in practice it requires the creation of an $N\times N$ attention matrix for every layer and attention head, which can consume large amounts of memory when applied to data types with many elements, like images or raw audio. A sampling of them are displayed here for a 128-layer model on CIFAR-10 images:

While many layers displayed sparse structure, some layers clearly display dynamic attention that stretch over the entirety of the image. In order to preserve the ability of our network to learn such patterns, we implemented a two-dimensional factorization of the attention matrix, where the network can attend to all positions through two steps of sparse attention.

Source: openai.com