Neural Machine Translation with Supervised Attention

By Lemao Liu, Masso Utiyama, Andrew M. Finch, Eiichiro Sumita. 2016

162 citations recorded by the source corpus.

The attention mechanism is appealing for neural machine translation, since it is able to dynamically encode a source sentence by generating a alignment between a target word and source words. Unfortunately, it has been proved to be worse than conventional alignment models in alignment accuracy. In this paper, we analyze and explain this issue from the point view of reordering, and propose a supervised attention which is learned with guidance from conventional alignment models. Experiments on two Chinese-to-English translation tasks show that the supervised attention mechanism yields better alignments leading to substantial gains over the standard attention based NMT.

Subfields: Artificial Intelligence, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Machine Translation

Citations in the corpus

Six Challenges for Neural Machine Translation (2017, background): “Note that the attention model may produce better word alignments by guided alignment training (Chen et al., 2016; Liu et al., 2016) where supervised word alignments (such as the ones produced by fast-align) are provided to model training.”
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, background): “Many works [15, 35, 16] suggest that attention weight assignment in encoder-decoder attention plays a role similar to word alignment in traditional approaches [1, 7, 27, 5].”
Learning to Deceive with Attention-Based Explanations (2019, background): “In some of these papers, the supervision comes from known word alignments for machine translation (Liu et al., 2016; Chen et al., 2016), or by aligning human eye-gaze with model’s attention for sequence classification (Barrett et al., 2018).”
Few-Shot Semantic Parsing for New Predicates (2021, methodology): “The prior work on supervised attention is not applicable, because it requires either large parallel data (Liu et al., 2016), significant manual effort (Bao et al., 2018; Rabi-novich et al., 2017), or it is designed only for applications other than semantic parsing (Liu et al., 2017; Kamigaito et…”
Attention in Natural Language Processing (2019, background): “in many scenarios, including machine translation [30], [35],”
What does Attention in Neural Machine Translation Pay Attention to? (2017, methodology): “The first model is a non-recurrent attention model which is equivalent to the “global attention” method proposed by Luong et al. (2015a). The second attention model that we use in our investigation is an input-feeding model similar to the attention model first proposed by Bahdanau et al.”
Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021, background): “In particular, word alignment plays a crucial role in many machine translation (MT) related methods, including guiding learned attention (Liu et al., 2016), incorporating lexicons during decoding (Arthur et al.”
Deriving Machine Attention from Human Rationales (2018, methodology): “Examples include using word alignments to learn attention in neural machine translation (Liu et al., 2016), employing argu-ment words to supervise attention in event detection (Liu et al., 2017), utilizing linguistically-motivated annotations to guide attention in constituency parsing (Kamigaito et…”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “Alternatively, inﬂexible models with strong inductive biases have in the past been used to guide (overly) ﬂexible neural models in various ways, for example by helping to generate additional data [58, 75] or inducing structures with which to regularize/augment models [24, 74, 3, 127].”
Neural Math Word Problem Solver with Reinforcement Learning (2018, background): “Some recent work have been proposed to improve the alignment for neural machine translation (Liu et al., 2016; Mi et al., 2016).”
Accelerating Neural Transformer via an Average Attention Network (2018, methodology): “Mi et al. (2016) and Liu et al. (2016) employ standard word alignment to supervise the automatically generated attention weights.”
Neural Machine Translation (2020, unknown): “Note that the attention model may produce better word alignments by guided alignment training (Chen et al., 2016b; Liu et al., 2016) where supervised word alignments (such as the ones produced by fast-align) are provided to model training.”
Evaluating Explanation Methods for Neural Machine Translation (2020, methodology): “Attention Since Bahdanau et al. (2014) propose the attention mechanism for NMT, it has been the most popular explanation method for NMT (Tu et al., 2016; Mi et al., 2016; Liu et al., 2016; Zenkel et al., 2019).”
Jointly Learning to Align and Translate with Transformer Models (2019, background): “Mi et al. (2016), Chen et al. (2016), Liu et al. (2016) and Alkhouli and Ney (2017) supervise the attention mechanisms of recurrent models (Bahdanau et al., 2015) in this way.”
SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings (2020, methodology): “Word alignments are essential for statistical machine translation and useful in NMT, e.g., for imposing priors on attention matrices (Liu et al., 2016; Alkhouli and Ney, 2017; Alkhouli et al., 2018) or for decoding (Alkhouli et al., 2016; Press and Smith, 2018).”
Saliency-driven Word Alignment Interpretation for Neural Machine Translation (2019, background): “There are also a few other studies that inject alignment supervision during NMT training ( Mi et al., 2016; Liu et al., 2016).”