'Attention Is All You Need' introduces the Transformer

In June 2017 eight researchers at Google published a paper titled 'Attention Is All You Need', introducing the Transformer, a neural network architecture based on a mechanism called attention, which lets a model weigh the relevance of every part of an input to every other part, and which can be trained in parallel on very large amounts of data. It was first used for machine translation. Transformers became the basis of large language models such as BERT and GPT, and were extended to images, audio and proteins. They are the foundation of the surge in generative AI of the 2020s.

Open on the timeline