What does Attention in Neural Machine Translation Pay Attention to?

By Hamidreza Ghader, Christof Monz. 2017

117 citations recorded by the source corpus.

Attention in neural machine translation provides the possibility to encode relevant parts of the source sentence at each translation step. As a result, attention is considered to be an alignment model as well. However, there is no work that specifically studies attention and provides analysis of what is being learned by attention models. Thus, the question still remains that how attention is similar or different from the traditional alignment. In this paper, we provide detailed analysis of attention and compare it to traditional alignment. We answer the question of whether attention is only capable of modelling translational equivalent or it captures more information. We show that attention is different from alignment in some cases and is capturing useful information other than alignments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, background): “There are several works analyzing attention weights of different NMT models (Ghader and Monz, 2017; Voita et al., 2018; Tang et al., 2018; Raganato and Tiedemann, 2018).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “, 2016a) RNN states/gates POS, syntactic role, gender, case, definiteness, verb form, mood Classification, correlation (Wu and King, 2016) RNN states/gates Acoustic features Correlation (Ghader and Monz, 2017) Attention weights POS, word alignment Distribution measures, match with alignments (Voita ”
Analyzing the Structure of Attention in a Transformer Language Model (2019, methodology): “We measure attention dispersion based on the entropy4 of the attention distribution (Ghader and Monz, 2017):”
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, methodology): “Many works [15, 35, 16] suggest that attention weight assignment in encoder-decoder attention plays a role similar to word alignment in traditional approaches [1, 7, 27, 5].”
Neural Machine Translation: A Review (2019, background): “Ghader and Monz (2017) provide a detailed analysis of how NMT uses attention to condition on the source sentence.”
Jointly Learning to Align and Translate with Transformer Models (2019, methodology): “…evaluate and analyze the quality of attention probabilities learnt by different parts of the Transformer model with respect to modeling alignments, which contributes to previous work on understanding attention mechanisms (Ghader and Monz, 2017; Raganato and Tiedemann, 2018; Tang et al., 2018).”
An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation (2018, background): “Koehn and Knowles (2017) and Ghader and Monz (2017) investigate the relation between attention mechanisms and the traditional word alignment.”
Saliency-driven Word Alignment Interpretation for Neural Machine Translation (2019, methodology): “While Ghader and Monz (2017) showed that the AER obtained by LSTM model is close to that of GIZA++, our experiments yield a much larger difference.”
Interrogating the Explanatory Power of Attention in Neural Machine Translation (2019, background): “Analyzing and interpreting the attention mechanism in NLP (Koehn and Knowles, 2017; Ghader and Monz, 2017; Tang and Nivre, 2018; Clark et al., 2019; Vig and Belinkov, 2019) is another direction that has drawn major interest.”
Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2020, background): “Second, several studies (Ghader and Monz, 2017; Li et al., 2019) have shown that attention may fail to capture word alignment for some language pairs, and Legrand et al. (2016); Alkhouli et al. (2018) proposed neural word alignment to improve the word alignment quality.”
SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings (2020, methodology): “With the rise of NMT (Bahdanau et al., 2014), attempts have been made to interpret attention matrices as soft word alignments (Koehn and Knowles, 2017; Ghader and Monz, 2017).”