On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
By Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, Yoshua Bengio. 2014
7,554 citations recorded by the source corpus.
Neural machine translation is a relatively new approach to statistical machine translation based purely on neural networks. The neural machine translation models often consist of an encoder and a decoder. The encoder extracts a fixed-length representation from a variable-length input sentence, and the decoder generates a correct translation from this representation. In this paper, we focus on analyzing the properties of the neural machine translation using two models; RNN Encoder--Decoder and a newly proposed gated recursive convolutional neural network. We show that the neural machine translation performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase. Furthermore, we find that the proposed gated recursive convolutional network learns a grammatical structure of a sentence automatically.
Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Luke S. Zettlemoyer
- Quoc V. Le
- Kyunghyun Cho
In Molecular Biology
- Matthias Fey
- Trevor Cohn
- David Chiang
- Tommi S. Jaakkola
- Kyunghyun Cho
- Yoshua Bengio
- Jonathan Berant
- Jure Leskovec
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Kyunghyun Cho
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yejin Choi
- Noah A. Smith
- Percy Liang
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Kyunghyun Cho
- Yoav Goldberg
- Ruslan Salakhutdinov
- Yonatan Belinkov
- Richard Socher
- Graham Neubig
Citations in the corpus
- Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “3 As a comparison, we also report the performance of an RNN Encoder–Decoder which was proposed recently by Cho et al. (2014a).”
- Deep Contextualized Word Representations (2018, background): “It adds a self-attention layer after the bidirectional attention component, simpli-fies some of the pooling operations and substitutes the LSTMs for gated recurrent units (GRUs; Cho et al., 2014).”
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “Our first, and simplest, encoders apply recurrent neural networks using either LSTM or GRU (Cho et al., 2014) modules, as in sequence to sequence encoders (Sutskever et al.”
- Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, methodology): “We consider three sentence encoding models: a simple 1024D sum-of-Words (CBOW) encoding, a 1024D GRU recurrent neural network (Cho et al., 2014), and a 512D bidirectional GRU RNN (BiGRU).”
- Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “For regularization, we add 25% dropout to the input of each GRU, but not to the recurrent connections.”
- Skip-Thought Vectors (2015, background): “In recent years, several approaches have been developed for learning composition operators that map word vectors to sentence vectors including recursive networks [1], recurrent networks [2], convolutional networks [3, 4] and recursive-convolutional methods [5, 6] among others.”
- The Curious Case of Neural Text Degeneration (2019, methodology): “Beam search has been used successfully in numerous research into non-open-ended generation tasks such as machine translation, data-to-text generation, and summarization (Bahdanau et al., 2015; Luong et al., 2015; Cho et al., 2014).”
- Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “Typically, an encoder-decoder architecture (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015) with a RNN-based decoder is used.”
- RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “To alleviate the difficulty of training RNNs due to vanishing and exploding gradients (Bengio et al., 1994), we use Gated Recurrent Units (GRUs) (Cho et al., 2014) which have been demonstrated to have good empirical performance (Chung et al., 2014; J´ozefowicz et al., 2015).”
- MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms (2019, methodology): “For our initial sequence-to-program model, we follow the attention-based NMT paradigm of (Bahdanau et al., 2015; Cho et al., 2014).”
- Learning with Latent Language (2017, methodology): “In all models, RNN encoders and decoders use gated recurrent units (Cho et al., 2014).”
- A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “Recurrent networks are typically parameterized as long short term memory networks (LSTM; Hochre-iter et al. 1997) or gated recurrent units (GRU; Cho et al. 2014), often with residual or skip connections (Wu et al., 2016; Zhou et al., 2016) to enable stacking of several layers ( 2).”
- Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “We use the GRU cell as the rationalizer.”