Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures

By Gongbo Tang, Mathias Müller, Annette Rios Gonzales, Rico Sennrich. 2018

276 citations recorded by the source corpus.

Recently, non-recurrent architectures (convolutional, self-attentional) have outperformed RNNs in neural machine translation. CNNs and self-attentional networks can connect distant words via shorter network paths than RNNs, and it has been speculated that this improves their ability to model long-range dependencies. However, this theoretical argument has not been tested empirically, nor have alternative explanations for their strong performance been explored in-depth. We hypothesize that the strong performance of CNNs and self-attentional networks could also be due to their ability to extract semantic features from the source text, and we evaluate RNNs, CNNs and self-attention networks on two tasks: subject-verb agreement (where capturing long-range dependencies is required) and word sense disambiguation (where semantic feature extraction is required). Our experimental results show that: 1) self-attentional networks and CNNs do not outperform RNNs in modeling subject-verb agreement over long distances; 2) self-attentional networks perform distinctly better than RNNs and CNNs on word sense disambiguation.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, background): “, 2018), word sense disambiguation (Tang et al., 2018), learning sentence representations (Devlin et al.”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “There are several works analyzing attention weights of different NMT models (Ghader and Monz, 2017; Voita et al., 2018; Tang et al., 2018; Raganato and Tiedemann, 2018).”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, background): “However, the ability of self-attention to model long-range dependencies has recently come into question (Tang et al., 2018) and the unlimited context size is computationally very challenging due to the quadratic complexity in the input length.”
Are Sixteen Heads Really Better than One? (2019, background): “It has been shown that beyond improving performance, MHA can help with subject-verb agreement (Tang et al., 2018) and that some heads are predictive of dependency structures (Raganato and Tiedemann, 2018).”
A Structural Probe for Finding Syntax in Word Representations (2019, background): “Recent work has analyzed model behavior to determine if a model understands hierarchy and other linguistic phenomena (Linzen, 2018; Gulordava et al., 2018; Kuncoro et al., 2018; Linzen and Leonard, 2018; van Schijndel and Linzen, 2018; Tang et al., 2018; Futrell et al., 2018).”
Assessing BERT's Syntactic Abilities (2019, methodology): “This differs from Linzen et al. (2016) and Gulordava et al. (2018) by considering the entire sentence (excluding the verb) and not just its preﬁx leading to the verb, and differs from Marvin and Linzen (2018) by conditioning the focus verb on bidirectional context.”
Revealing the Dark Secrets of BERT (2019, neutral): “Their ﬁndings suggest that (a) the middle layers of Transformer-based architectures are the most transferable to other tasks, and (b) higher layers of Transformers are not as task speciﬁc as the ones of RNNs. Tang et al. (2018) argued that models using self-attention outperform CNN-and RNN-based mod”
Linguistic Knowledge and Transferability of Contextual Representations (2019, background): “Better understanding the differences between transformers and LSTMs is an active area of research (Chen et al., 2018; Tang et al., 2018), and we leave further exploration of these observations to future work.”
Adaptively Sparse Transformers (2019, background): “As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens (Tang et al., 2018; Ra-ganato and Tiedemann, 2018; Mareˇcek and Rosa, 2018; Tenney et al., 2019; Voita et al., 2019).”
Augmenting Neural Machine Translation with Knowledge Graphs (2019, methodology): “We follow the notations from Tang et al. (2018b) in the subsequent sections: hli corresponds to the hidden state at step i of layer l. hli−1 represents the hidden state at the previous step of layer l while hl−1i means the hidden state at i of l− 1 layer.”
Effective Sentence Scoring Method Using BERT for Speech Recognition (2019, background): “Although most language models are based on RNN, self-attention network (SAN) LMs have recently shown competitive performance on sequence modeling with a slight tradeoff between speed and accuracy Peters et al. (2018b); Tang et al. (2018).”
Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study (2019, background): “For example, Tran et al. (2018) ﬁnd that recurrent models are better at modeling hierarchical structure while Tang et al. (2018) ﬁnd that feedfor-ward architectures like the transformer and convolutional models are not better than RNNs at modeling long-distance agreement.”
BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer (2019, result): “This phenomenon is consistent with the empirical result in [48] that large h is essential for capturing long distance dependencies with multi-head self-attention.”
Saliency-driven Word Alignment Interpretation for Neural Machine Translation (2019, background): “Zenkel et al. (2019); Tang et al. (2018b); Raganato and Tiedemann (2018) pointed out that the attention-induced alignment is particularly noisy with Transformer models.”
From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions (2019, background): “Tang et al. (2018) show the Transformer networks perform better ar X iv :1 90 6.”
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020, background): “The first line of research focuses on the interpretation of the network, in particular on the analysis of attention mechanisms and the interpretability of the weights and connections (Raganato and Tiedemann, 2018; Tang et al., 2018; Mareček and Rosa, 2019; Voita et al., 2019a; Brunner et al., 2020).”
An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation (2018, methodology): “Tang et al. (2018b) have shown that Transformer models are better than RNN models in WSD because of their stronger encoding ability.”
CNNs found to jump around more skillfully than RNNs: Compositional Generalization in Seq2seq Convolutional Networks (2019, background): “Tang et al. (2018) have shown that RNNs and selfattentive models are better than CNNs at capturing long-distance agreement, while self-attentive networks excel at word sense disambiguation.”