Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
By Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio. 2014
26,848 citations recorded by the source corpus.
In this paper, we propose a novel neural network model called RNN Encoder‐ Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixedlength vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder‐Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Graham Neubig
- Kyunghyun Cho
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Kyunghyun Cho
- Luke S. Zettlemoyer
- Percy Liang
- Richard Socher
- Yejin Choi
Citations in the corpus
- Attention is All you Need (2017, background): “Recurrent neural networks, long short-term memory [12] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [29, 2, 5].”
- Convolutional Sequence to Sequence Learning (2017, background): “Popular choices for recurrent networks in encoder-decoder models are long short term memory networks (LSTM; Hochreiter & Schmidhuber, 1997) and gated recurrent units (GRU; Cho et al., 2014).”
- Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “Neural Machine Translation (NMT) (Kalchbren-ner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014) is an end-to-end approach to automated translation.”
- Neural Machine Translation in Linear Time (2016, methodology): “In our comparison we consider the following neural translation models: the Recurrent Continuous Translation Model (RCTM) 1 and 2 (Kalchbrenner & Blunsom, 2013); the RNN Enc-Dec (Sutskever et al., 2014; Cho et al., 2014); the RNN Enc-Dec Att with the attentional pooling mechanism (Bahdanau et al., 20”
- Can Active Memory Replace Attention? (2016, background): “While a pure sequence-to-sequence recurrent neural network can already obtain good translation results [4, 6], it suffers from the fact that the whole sentence to be translated needs to be encoded into a single fixed-size vector.”
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “In practice, however, NMT systems used to be worse in accuracy than phrase-based translation systems, especially when training on very large-scale datasets as used for the very best publicly available translation systems.”
- Layer Normalization (2016, methodology): “We refer the reader to the appendix for a description of how layer normalization is applied to GRU.”
- Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, neutral): “For complete comparison with previous work on English-to-French task, we also show the results with small vocabulary of 30K input words and 30K output words on the sub train set with selected 12M parallel sequences (Schwenk, 2014; Sutskever et al., 2014; Cho et al., 2014).”
- Exploring the Limits of Language Modeling (2016, background): “…recently proposed sequence-to-sequence models employ conditional language models (Mikolov & Zweig, 2012) as their key component to solve diverse tasks like machine translation (Sutskever et al., 2014; Cho et al., 2014; Kalch-brenner et al., 2014) or video generation (Srivastava et al., 2015a).”
- Neural GPUs Learn Algorithms (2015, methodology): “A GRU is similar to an LSTM, but its input and state are the same size, which makes it easier for us to generalize it later; a highway network could have also been used (Srivastava et al., 2015), but it lacks the reset gate.”
- Neural Machine Translation of Rare Words with Subword Units (2015, background): “The encoder is a bidirectional neural network with gated recurrent units (Cho et al., 2014) that reads an input sequence x = ( x 1 , ..., x m ) and calculates a forward sequence of hidden states ( −→ h 1 , ..., −→ h m ) , and a backward sequence ←− −→ ←− The hidden states h j and h j are…”
- Effective Approaches to Attention-based Neural Machine Translation (2015, methodology): “cent NMT work such as (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Jean et al., 2015) have in common.”
- Sequence to Sequence Learning with Neural Networks (2014, methodology): “[5] used an LSTM-like RNN architecture to map sentences into vectors and back, although their primary focus was on integrating their neural network into an SMT system.”
- Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “3 As a comparison, we also report the performance of an RNN Encoder–Decoder which was proposed recently by Cho et al. (2014a).”
- Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “Much of the progress in this area has been made by mitigating the vanishing gradients problem [14] by architectures such as LSTMs [15], GRU [3], Recurrent Highway Networks [51], Uni-tary RNNs [1] and others.”
- Neural Network Acceptability Judgments (2018, background): “These models are widely used to encode features of sentences in fixed-length sentence embeddings (Cho et al., 2014; Sutskever et al., 2014; Kiros et al., 2015).”