How Transformers Work – Model Used by Open AI and DeepMind

Internally, a LSTM looks like the following:
Each cell takes as inputs x_t (a word in the case of a sentence to sentence translation), the previous cell state and the output of the previous cell. To summarize, LSTMs and RNNs present 3 problems:
Sequential computation inhibits parallelization
No explicit modeling of long and short range dependencies
“Distance” between positions is linear

Attention
To solve some of these problems, researchers created a technique for paying attention to specific words. Wavenet, model is a Convolutional Neural Network (CNN)The reason why Convolutional Neural Networks can work in parallel, is that each word on the input can be processed at the same time and does not necessarily depend on the previous words to be translated.

Source: medium.com