Neural Machine Translation by Jointly Learning to Align and Translate
By Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. International Conference on Learning Representations, 2014
29,733 citations recorded by the source corpus.
Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures
View Paper: https://arxiv.org/abs/1409.0473
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Percy Liang
- Luke S. Zettlemoyer
- Quoc V. Le
- Graham Neubig
- Kyunghyun Cho
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Percy Liang
- Yejin Choi
- Richard Socher
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Kyunghyun Cho
- Ruslan Salakhutdinov
- Mohit Bansal
- Jianfeng Gao
- Yoav Goldberg
- Richard Socher
Citations in the corpus
- Attention is All you Need (2017, methodology): “Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 29].”
- A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “Attention mechanisms (Bahdanau et al., 2014) make these models more performant and scalable, allowing them to look back at parts of the encoded input sequence while the output is generated.”
- Convolutional Sequence to Sequence Learning (2017, methodology): “The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”
- Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “Our models are based on an encoder-decoder architecture with attention mechanism (Bahdanau et al., 2015; Luong et al., 2015a), as shown in fig-ure 1.”
- Structured Attention Networks (2017, background): “For example, consider the case of attention-based neural machine translation (Bahdanau et al., 2015).”
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014).”
- Neural Machine Translation in Linear Time (2016, methodology): “…have been proposed for modelling translation, such as encoder-decoder networks (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Kaiser & Bengio, 2016), networks with attentional pooling (Bahdanau et al., 2014) and two-dimensional networks (Kalchbrenner et al., 2016a).”
- Can Active Memory Replace Attention? (2016, methodology): “This is the same task that was used to introduce attention [5], but – to avoid the problem with the UNK token – we spell-out each word that is not in the vocabulary.”
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “While the translation accuracy of these systems has been encouraging, systematic comparison with large scale, production quality phrase-based translation systems has been lacking.”
- Using the Output Embedding to Improve Language Models (2016, background): “In neural machine translation (NMT) models (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2014), the decoder, which generates the translation of the input sentence in the target language, is a language model that is conditioned on both the previous words…”
- Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, methodology): “Note that RNNsearch-LV uses a large output vocabulary of 500K words based on the standard attention model RNNsearch.”
- Recurrent Neural Network Grammars (2016, neutral): “…state-ofthe-art baselines have been obtained using RNNs (Zaremba et al., 2015; Mikolov et al., 2010) as well as in various conditional language modeling tasks such as machine translation (Bahdanau et al., 2015), image caption generation (Xu et al., 2015), and dialogue generation (Wen et al., 2015).”
- Long Short-Term Memory-Networks for Machine Reading (2016, methodology): “In the encoder-decoder architecture, this problem can be sidestepped with an attention mechanism to learn a soft alignment between the encoding and decoding states (Bahdanau et al., 2014).”
- Neural GPUs Learn Algorithms (2015, methodology): “…networks with such cells can learn very complex tasks in an end-to-end manner, such as translation (Sutskever et al., 2014; Bahdanau et al., 2014; Cho et al., 2014), parsing (Vinyals & Kaiser et al., 2015), speech recognition (Chan et al., 2016) or image caption generation (Vinyals et al., 2014).”
- Multi-task Sequence to Sequence Learning (2015, methodology): “A criticism of our work is that our sequence to sequence models do not employ the attention mechanism (Bahdanau et al., 2015).”
- Effective Approaches to Attention-based Neural Machine Translation (2015, methodology): “On the other hand, in (Bahdanau et al., 2015; Jean et al., 2015) and this work, s, in fact, implies a set of source hidden states which are consulted throughout the entire course of the translation process.”
- Grammar as a Foreign Language (2014, methodology): “We adapted the attention model from [2] which, to produce each output symbolBt, uses an attention mechanism over the encoder LSTM states.”
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, background): “[2014] and Bahdanau et al. [2014] reported that recurrent neural networks are able to perform as well as the existing, well-developed systems on a challenging task of machine translation.”
- Sequence to Sequence Learning with Neural Networks (2014, methodology): “We were initially convinced that the LSTM would fail on long sentences due to its limited memory, and other researchers reported poor performance on long sentences with a model similar to ours [5, 2, 26].”