Attention is All you Need
By Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. Neural Information Processing Systems, 2017
185,508 citations recorded by the source corpus.
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures
View Paper: https://arxiv.org/abs/1706.03762
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Percy Liang
- Luke S. Zettlemoyer
- Quoc V. Le
- Graham Neubig
- Sergey Levine
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Percy Liang
- Kyunghyun Cho
- Noah A. Smith
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Jianfeng Gao
- Alan L. Yuille
- Ruslan Salakhutdinov
- Kyunghyun Cho
- Mohit Bansal
- Dacheng Tao
Citations in the corpus
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “For example, the largest Transformer explored in Vaswani et al. (2017) is (L=6, H=1024, A=16) with 100M parameters for the encoder, and the largest Transformer we have found in the literature is (L=64, H=512, A=2) with 235M parameters (Al-Rfou et al., 2018).”
- U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, background): “The self-attention layer is constructed the same way as (Vaswani et al. 2017): where (cid:98) A is the representation after self-attention of the fused information A .”
- Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “The Transformer, introduced by Vaswani et al. (2017), is a feed forward self-attention based architecture.”
- Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “For more details on the transformer architecture, refer to [42] and the tensor2tensor library 2 .”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “…by methods like soft-attention, is instrumental in achieving state-of-the-art performance on tasks such as machine translation (Bah-danau et al., 2014; Vaswani et al., 2017), question answering (Seo et al., 2016; Xiong et al., 2016), and natural language inference (Rockt¨aschel et al., 2016) .”
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “For the self-attention-layer, we adopt the multi-head attention mechanism defined in (Vaswani et al., 2017a) which, for each position in the input, called the query, computes a weighted sum of all positions, or keys, in the input based on the similarity between the query and key as measured by the…”
- MaskGAN: Better Text Generation via Filling in the ______ (2018, background): “Given this, we suspect another promising avenue would be to consider GAN-training with attention-only models as in Vaswani et al. (2017).”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Early results on transfer learning for NLP leveraged recurrent neural networks [Peters et al., 2018; Howard and Ruder, 2018], but it has recently become more common to use models based on the “Transformer” architecture [Vaswani et al., 2017].”
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “BERT uses the now ubiquitous transformer architecture (Vaswani et al., 2017), which we will not review in detail.”
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “where Q, K, V denote the query, key, and value in an attention operation [33].”
- Scaling Laws for Neural Language Models (2020, methodology): “In this work we will empirically investigate the dependence of language modeling loss on all of these factors, focusing on the Transformer architecture [VSP + 17, LSP + 18].”
- Training Compute-Optimal Large Language Models (2022, positive): “These large autoregressive transformers (Vaswani et al., 2017) have demonstrated impressive performance on many tasks using a variety of evaluation protocols such as zero-shot, few-shot, and fine-tuning.”
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Like GPT-3, these models are all variants of the Transformer architecture (Vaswani et al., 2017).”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “Following recent work on large language models, our network is based on the transformer architecture (Vaswani et al., 2017).”
- Mistral 7B (2023, background): “Mistral 7B is based on a transformer architecture [27].”
- Mixtral of Experts (2024, methodology): “Mixtral is based on a transformer architecture [31] and uses the same modifications as described in [18], with the notable exceptions that Mix-tral supports a fully dense context length of 32k tokens, and the feed-forward blocks are replaced by Mixture-of-Expert layers (Section 2.1).”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “We use BART-large [28], a pre-trained seq2seq transformer [52] with 400M parameters.”
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, positive): “Self-attention-based architectures, in particular Transformers (Vaswani et al., 2017), have become the model of choice in natural language processing (NLP).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Vision Transformer (ViT) We also include four ViT (Dosovitskiy et al., 2020) checkpoints pretrained on the ImageNet-21k dataset, namely ViT-B/32, ViT-B/16, ViTL/16, and ViT-H/14.”