An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation

By Gongbo Tang, Rico Sennrich, Joakim Nivre. 2018

89 citations recorded by the source corpus.

Recent work has shown that the encoder-decoder attention mechanisms in neural machine translation (NMT) are different from the word alignment in statistical machine translation. In this paper, we focus on analyzing encoder-decoder attention mechanisms, in the case of word sense disambiguation (WSD) in NMT models. We hypothesize that attention mechanisms pay more attention to context tokens when translating ambiguous words. We explore the attention distribution patterns when translating ambiguous nouns. Counterintuitively, we find that attention mechanisms are likely to distribute more attention to the ambiguous noun itself rather than context tokens, in comparison to other nouns. We conclude that attention is not the main mechanism used by NMT models to incorporate contextual information for WSD. The experimental results suggest that NMT models learn to encode contextual information necessary for WSD in the encoder hidden states. For the attention mechanism in Transformer models, we reveal that the first few layers gradually learn to “align” source and target tokens and the last few layers learn to extract features from the related but unaligned context tokens.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, methodology): “Many works [15, 35, 16] suggest that attention weight assignment in encoder-decoder attention plays a role similar to word alignment in traditional approaches [1, 7, 27, 5].”
On Identifiability in Transformers (2019, background): “…in prior work (Vaswani et al., 2017; Vig, 2019; Marecek & Rosa, 2018; Clark et al., 2019; Raganato & Tiedemann, 2018; Voita et al., 2019; Tang et al., 2018; Wangperawong, 2018; Padigela et al., 2019; Baan et al., 2019; Lin et al., 2019; Dehghani et al., 2019; Zenkel et al., 2019; Klein…”
It’s not a Non-Issue: Negation as a Source of Error in Machine Translation (2020, unknown): “…anaphora resolution (Hard-meier et al., 2014; Voita et al., 2018) and pronoun translation (Guillou and Hardmeier, 2016), modality (Baker et al., 2012), ellipsis and deixis (Voita et al., 2019), word sense disambiguation (Tang et al., 2018), and morphological competence (Bur-lot and Yvon, 2017).”
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model (2019, background): “…meaning is represented in attention-based networks or contextual embeddings include “probes” (learned models such as classiﬁers) as part of their evaluation setup to ‘extract’ information from the embeddings (Peters et al., 2018; Tang et al., 2018; Coenen et al., 2019; Chang and Chen, 2019, e.g.).”
Neural Machine Translation: A Review (2019, background): “Some ablation studies in the literature aim to factor out or explain the contributions of these different techniques [123, 124, 366, 396].”
Disentangled Non-Local Neural Networks (2020, background): “Our work is also re-lated to several approaches that analyze the non-local/self-attention mechanism in depth, including the performance of individual terms [21,32,46] on various tasks.”
Gender Bias in Multilingual Neural Machine Translation: The Architecture Matters (2020, methodology): “Previous works have studied the relationship between the attention mechanism and the model output (Clark et al., 2019; Kovaleva et al., 2019; Reif et al., 2019; Lin et al., 2019; Mareček and Rosa, 2019; Htut et al., 2019; Raganato and Tiedemann, 2018; Tang et al., 2018).”
Augmenting Neural Machine Translation with Knowledge Graphs (2019, methodology): “We follow the notations from Tang et al. (2018b) in the subsequent sections: hli corresponds to the hidden state at step i of layer l. hli−1 represents the hidden state at the previous step of layer l while hl−1i means the hidden state at i of l− 1 layer.”
Saliency-driven Word Alignment Interpretation for Neural Machine Translation (2019, methodology): “We follow Zenkel et al. (2019) in data setup and use the accompanied scripts of that paper4 for preprocessing.”