Effective Approaches to Attention-based Neural Machine Translation
By Thang Luong, Hieu Pham, Christopher D. Manning. 2015
8,424 citations recorded by the source corpus.
An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout. Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT’15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker. 1
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Luke S. Zettlemoyer
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Kyunghyun Cho
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Natural Language Processing
- Graham Neubig
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Yejin Choi
- Percy Liang
- Richard Socher
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Kyunghyun Cho
- Yoav Goldberg
- Richard Socher
- Jianfeng Gao
- Mohit Bansal
- Graham Neubig
Citations in the corpus
- Attention is All you Need (2017, background): “Numerous efforts have since continued to push the boundaries of recurrent language models and encoder-decoder architectures [38, 24, 15].”
- Convolutional Sequence to Sequence Learning (2017, methodology): “The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”
- Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “Our models are based on an encoder-decoder architecture with attention mechanism (Bahdanau et al., 2015; Luong et al., 2015a), as shown in fig-ure 1.”
- Structured Attention Networks (2017, background): “The baseline NMT system is from Luong et al. (2015).”
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a).”
- Neural Machine Translation in Linear Time (2016, methodology): “In our comparison we consider the following neural translation models: the Recurrent Continuous Translation Model (RCTM) 1 and 2 (Kalchbrenner & Blunsom, 2013); the RNN Enc-Dec (Sutskever et al., 2014; Cho et al., 2014); the RNN Enc-Dec Att with the attentional pooling mechanism (Bahdanau et al., 20”
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “…alignment model [31], or by using the attention mechanism to copy rare words [37], these approaches are both unreliable at scale, since the quality of the alignments varies across languages, and the latent alignments produced by the attention mechanism are unstable when the network is deep.”
- Neural Machine Translation of Rare Words with Subword Units (2015, methodology): “Regarding other neural systems, Luong et al. (2015a) report a B LEU score of 25.9 on newstest2015, but we note that they use an ensemble of 8 independently trained models, and also report strong improvements from applying dropout, which we did not use.”
- Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “The encoder’s representations are passed into an LSTM decoder using a bilinear attention mechanism (Luong et al., 2015).”
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “In this work, we consider attention-based neural machine translation (NMT) models Bahdanau et al. (2015); Luong et al. (2015), which have demonstrated excellent translation quality Wu et al. (2016), as the core models of our data augmentation pipeline.”
- MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “With these language models, we now pretrain the seq2seq model on the in-filling task using maximum likelihood, in particular, the attention parameters as described in Luong et al. (2015).”
- Learned in Translation: Contextualized Word Vectors (2017, background): “Luong et al. [2015] further study the effectiveness of various attention mechanisms with respect to machine translation.”
- The Curious Case of Neural Text Degeneration (2019, methodology): “Beam search has been used successfully in numerous research into non-open-ended generation tasks such as machine translation, data-to-text generation, and summarization (Bahdanau et al., 2015; Luong et al., 2015; Cho et al., 2014).”
- The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “First, the LSTM produces an intermediate state ht using the previous answer word At−1self and recurrent context state [Luong et al., 2015b]: LSTM([(Aself )t−1 ; c̃t−1], ht−1) = ht ∈ R d (7) Context and Question Attention.”
- Deep Learning Scaling is Predictable, Empirically (2017, methodology): “To test NMT, we train a SOTA sequence-to-sequence model with global attention (Luong et al. (2015)) on the 2016 Conference on Machine Translation (WMT’16) German-to-English data set.”
- Sequence-Level Knowledge Distillation (2016, background): “Encoder In attention-based models (Bahdanau et al., 2015; Luong et al., 2015), the encoder reads the source sentence and outputs a sequence of vectors (one vector for each time step) to be attended to during decoding.”
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018, background): “Neural machine translation (NMT) (Bahdanau et al., 2014; Luong et al., 2015; Wu et al., 2016; Vaswani et al., 2017) has especially gained increasing popularity, as it can leverage neural networks to directly perform translations with a simple end-to-end architecture.”