Sequence to Sequence Learning with Neural Networks
By Ilya Sutskever, Oriol Vinyals, Quoc V. Le. Neural Information Processing Systems, 2014
22,211 citations recorded by the source corpus.
Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures
View Paper: https://arxiv.org/abs/1409.3215
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Graham Neubig
- Kyunghyun Cho
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Yejin Choi
- Percy Liang
- Richard Socher
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Ruslan Salakhutdinov
- Kyunghyun Cho
- Jianfeng Gao
- Mohit Bansal
- Richard Socher
- Alan L. Yuille
Citations in the corpus
- Attention is All you Need (2017, background): “Recurrent neural networks, long short-term memory [12] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [29, 2, 5].”
- A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “Neural encoder-decoder models are widely used in NLP applications such as machine translation (Sutskever et al., 2014), summarization (Chopra et al., 2016; Nallapati et al., 2016), and question answering (Hermann et al., 2015).”
- Convolutional Sequence to Sequence Learning (2017, background): “Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al.”
- Massive Exploration of Neural Machine Translation Architectures (2017, background): “Neural Machine Translation (NMT) (Kalchbren-ner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014) is an end-to-end approach to automated translation.”
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015).”
- Neural Machine Translation in Linear Time (2016, methodology): “A number of neural architectures have been proposed for modelling translation, such as encoder-decoder networks (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Kaiser & Bengio, 2016), networks with attentional pooling (Bahdanau et al., 2014) and two-dimensional networks…”
- Can Active Memory Replace Attention? (2016, methodology): “While a pure sequence-to-sequence recurrent neural network can already obtain good translation results [4, 6], it suffers from the fact that the whole sentence to be translated needs to be encoded into a single fixed-size vector.”
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “While the translation accuracy of these systems has been encouraging, systematic comparison with large scale, production quality phrase-based translation systems has been lacking.”
- Layer Normalization (2016, neutral): “The recent sequence to sequence models [Sutskever et al., 2014] utilize compact recurrent neural networks to solve sequential prediction problems in natural language processing.”
- Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, methodology): “As a reference, previous work showed that oracle rescoring of the 1000-best sequences generated by the SMT model can achieve the BLEU score of about 45 (Sutskever et al., 2014).”
- Exploring the Limits of Language Modeling (2016, background): “…recently proposed sequence-to-sequence models employ conditional language models (Mikolov & Zweig, 2012) as their key component to solve diverse tasks like machine translation (Sutskever et al., 2014; Cho et al., 2014; Kalch-brenner et al., 2014) or video generation (Srivastava et al., 2015a).”
- Neural GPUs Learn Algorithms (2015, methodology): “The learning of algorithms with neural networks has seen a lot of interest after the success of sequence-to-sequence neural networks on language processing tasks (Sutskever et al., 2014; Bahdanau et al., 2014; Cho et al., 2014).”
- Multi-task Sequence to Sequence Learning (2015, methodology): “The early seq2seq work (Sutskever et al., 2014; Cho et al., 2014; Luong et al., 2015b; Vinyals et al., 2015b) uses only the last encoder state to initialize the decoder and sets s = [ ] in Eq.”
- Effective Approaches to Attention-based Neural Machine Translation (2015, methodology): “As shown in Table 1, we achieve progressive improvements when (a) reversing the source sentence, +1.3 BLEU, as proposed in (Sutskever et al., 2014) and (b) using dropout, +1.4 BLEU.”
- Grammar as a Foreign Language (2014, methodology): “[1] introduced a neural network model for solving the general sequenceto-sequence problem, and Bahdanau et al.”
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, background): “Those tasks include, but are not limited to, speech recognition [see, e.g., Graves et al., 2013] and machine translation [see, e.g., Sutskever et al., 2014, Bahdanau et al., 2014].”
- Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “, 2014a) and (Sutskever et al., 2014) to encode a variable-length source sentence into a fixed-length vector and to decode the vector into a variable-length target sentence.”
- Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “We use an encoder-decoder sequence-to-sequence model with attention (Sutskever et al., 2014; Bahdanau et al., 2015).”