How Much Attention Do You Need? A Granular Analysis of Neural Machine Translation Architectures

By Tobias Domhan. 2018

79 citations recorded by the source corpus.

With recent advances in network architectures for Neural Machine Translation (NMT) recurrent models have effectively been replaced by either convolutional or self-attentional approaches, such as in the Transformer. While the main innovation of the Transformer architecture is its use of self-attentional layers, there are several other aspects, such as attention with multiple heads and the use of many attention layers, that distinguish the model from previous baselines. In this work we take a fine-grained look at the different architectures for NMT. We introduce an Architecture Definition Language (ADL) allowing for a flexible combination of common building blocks. Making use of this language we show in experiments that one can bring recurrent and convolutional models very close to the Transformer performance by borrowing concepts from the Transformer architecture, but not using self-attention. Additionally, we find that self-attention is much more important on the encoder side than on the decoder side, where it can be replaced by a RNN or CNN without a loss in performance in most settings. Surprisingly, even a model without any target side self-attention performs well.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Felix Stahlberg
4. Yingce Xia
5. Yoav Goldberg
6. Xu Tan
7. Fei Tian
8. Rico Sennrich

In Computer Vision and Pattern Recognition

1. Tao Qin
2. Tie-Yan Liu
3. Felix Stahlberg
4. Yingce Xia
5. Fei Tian
6. Luc Van Gool
7. Neil Houlsby
8. Dacheng Tao

In Natural Language Processing

1. Tao Qin
2. Tie-Yan Liu
3. Felix Stahlberg
4. Yingce Xia
5. Fei Tian
6. Yoav Goldberg
7. Rico Sennrich
8. Jingbo Zhu

Citations in the corpus

• Learning Deep Transformer Models for Machine Translation (2019, methodology): “In recent implementations (Klein et al., 2017; Vaswani et al., 2018; Domhan, 2018), layer normalization is applied to the input of every sub-layer (see Figure 1(b)): Eq.”
• Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures (2018, methodology): “Domhan (2018) applies these advanced techniques of Transformer models to both RNN and CNN models in Sockeye , minimizing the architectural difference between them.”
• Attention in Natural Language Processing (2019, background): “tion [37], [42], [48], [132], argumentation mining [125], text”
• Neural Machine Translation: A Review (2019, background): “Some ablation studies in the literature aim to factor out or explain the contributions of these different techniques [123, 124, 366, 396].”
• From Sparse to Soft Mixtures of Experts (2023, methodology): “Thus, when using pre-normalization as most modern Transformer architectures (Domhan, 2018; Xiong et al., 2020; Riquelme et al., 2021; Fedus et al., 2022), the inputs to the MoE layer are “layer normalized”.”
• An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation (2018, background): “More recently, Domhan (2018) has shown that multi-layer attention is crucial in NMT models.”
• Searching Better Architectures for Neural Machine Translation (2020, background): “There are also recent works trying to combine these different architectures [21]– [23].”