Recurrent Continuous Translation Models

By Nal Kalchbrenner, Phil Blunsom. 2013

1,476 citations recorded by the source corpus.

We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of stateof-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Massive Exploration of Neural Machine Translation Architectures (2017, background): “Using a total of more than 250,000 GPU hours, we explore common variations of NMT architectures and provide insight into which architectural choices matter most.”
Neural Machine Translation in Linear Time (2016, methodology): “A number of neural architectures have been proposed for modelling translation, such as encoder-decoder networks (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Kaiser & Bengio, 2016), networks with attentional pooling (Bahdanau et al., 2014) and two-dimensional networks…”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “In practice, however, NMT systems used to be worse in accuracy than phrase-based translation systems, especially when training on very large-scale datasets as used for the very best publicly available translation systems.”
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, background): “Neural machine translation (NMT) has attracted a lot of interest in solving the machine translation (MT) problem in recent years (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015).”
Multi-task Sequence to Sequence Learning (2015, background): “Recently, sequence to sequence ( seq2seq ) learning, proposed by Kalchbrenner & Blunsom (2013), Sutskever et al. (2014), and Cho et al. (2014), emerges as an effective paradigm for dealing with variable-length inputs and outputs. seq2seq learning, at its core, uses recurrent neural networks to map…”
Effective Approaches to Attention-based Neural Machine Translation (2015, methodology): “In (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Luong et al., 2015), the source representation s is only used once to initialize the decoder hidden state.”
Grammar as a Foreign Language (2014, methodology): “Kalchbrenner & Blunsom (2013) used a convolutional neural network to encode a variable-sized input sentence into a vector of a fixed dimension and used an RNN to produce the output sentence.”
Sequence to Sequence Learning with Neural Networks (2014, methodology): “Our work is closely related to Kalchbrenner and Blunsom [18], who were the first to map the input sentence into a vector and then back to a sentence, although they map sentences to vectors using convolutional neural networks, which lose the ordering of the words.”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “Neural machine translation is a newly emerging approach to machine translation, recently proposed by Kalchbrenner and Blunsom (2013), Sutskever et al. (2014) and Cho et al. (2014b).”
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “The closest approach related to the proposed RNN Encoder–Decoder is the Recurrent Continuous Translation Model (Model 2) proposed in (Kalchbrenner and Blunsom, 2013).”
Semi-supervised sequence tagging with bidirectional language models (2017, neutral): “Recently, neural LMs (Bengio et al., 2003; Mikolov et al., 2010) have also been integrated in neural machine translation systems (e.g., Kalchbrenner and Blunsom, 2013; Devlin et al., 2014) to score candidate translations.”
Skip-Thought Vectors (2015, methodology): “Several choices of encoder-decoder pairs have been explored, including ConvNet-RNN [10], RNN-RNN [11] and LSTM-LSTM [12].”
Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “To construct an encoder, we use a recurrent neural network, inspired by the success of encoder-decoder models for neural machine translation [12, 3, 1, 35].”
Massively Multilingual Neural Machine Translation (2019, background): “Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2014; Sutskever et al., 2014) is the current state-ofthe-art approach for machine translation in both academia (Bojar et al.”
Sequence-Level Knowledge Distillation (2016, methodology): “Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015) is a deep learningbased method for translation that has recently shown promising results as an alternative to statistical ap- proaches.”
Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “Recent developments in deep learning have led to signiﬁcantly improved quality on Neural Machine Translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017).”