Joint Language and Translation Modeling with Recurrent Neural Networks

By Michael Auli, Michel Galley, Chris Quirk, Geoffry Zweig. 2013

295 citations recorded by the source corpus.

We present a joint language and translation model based on a recurrent neural network which predicts target words based on an unbounded history of both source and target words. The weaker independence assumptions of this model result in a vastly larger search space compared to related feedforward-based language or translation models. We tackle this issue with a new lattice rescoring algorithm and demonstrate its effectiveness empirically. Our joint model builds on a well known recurrent neural network language model (Mikolov, 2012) augmented by a layer of additional inputs from the source language. We show competitive accuracy compared to the traditional channel model features. Our best results improve the output of a system trained on WMT 2012 French-English data by up to 1.5 BLEU, and by 1.1 BLEU on average across several test sets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Sequence to Sequence Learning with Neural Networks (2014, background): “[1], who combine an NNLM with a topic model of the input sentence, which improves rescoring performance.”
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “More recently, another encoder–decoder model using an RNN was proposed in (Auli et al., 2013), where the decoder is conditioned on a representation of either a source sentence or a source context.”
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models (2015, methodology): “…probabilistic models using word perplexity has shown promising results in several machine learning tasks including statistical machine translation (Auli et al. 2013; Sutskever, Vinyals, and Le 2014; Bahdanau, Cho, and Bengio 2015), speech recognition (Hinton et al. 2012; Deng and Li 2013) and…”
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses (2015, background): “For example, Mikolov and Zweig (2012) and Auli et al. (2013) use a pre-trained topic model.”
Fast and Robust Neural Network Joint Models for Statistical Machine Translation (2014, methodology): “Auli et al. (2013) use a ﬁxed continuous-space source representation, obtained from LDA (Blei et al., 2003) or a source-only NNLM.”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “In particular, many of the tasks and datasets available are constrained to narrow domains.”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “This template is naturally suited for many NLP tasks such as language modeling [2, 64, 65], machine translation [66, 67, 68], speech recognition [69, 70, 71, 72], image captioning [73].”
Sequence-to-sequence neural net models for grapheme-to-phoneme conversion (2015, positive): “In recent work on sequence to sequence translation, it has been shown that side-conditioned neural networks can be effectively used for both machine translation [1–6] and image captioning [7–10].”
Learning to Diagnose with LSTM Recurrent Neural Networks (2015, background): “Recurrent Neural Networks (RNNs), in particular those based on Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997), model varying-length sequential data, achieving state-of-the-art results for problems spanning natural language processing, image captioning, handwriting recognition, and ge”
Learning Continuous Phrase Representations for Translation Modeling (2014, methodology): “…the CPTM is learning the continuous representation of a phrase that is effective for SMT. Motivated by recent studies on continuous-space language models (e.g., Bengio et al. 2003; Mikolov et al. 2011; Schwenk et al., 2012), we use a neural network to project a word vector to a feature vector.”
Language Models for Image Captioning: The Quirks and What Works (2015, neutral): “The PPLX is in many ways the natural measure of a statistical LM, but can be loosely correlated with BLEU (Auli et al., 2013).”
Chinese Poetry Generation with Recurrent Neural Networks (2014, methodology): “…et al., 2010) with an auxiliary input layer, i.e., the context vector from the RCM. Similar strategies for encoding additional information have been adopted in related language modeling and machine translation work (Mikolov and Zweig, 2012; Kalchbrenner and Blunsom, 2013; Auli et al., 2013).”
A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “Works using the reranking approach include (Socher et al., 2013; Auli et al., 2013; Le & Zuidema, 2014; Zhu et al., 2015a) MEMM and hybrid approaches Other formulations are, of course, also possible.”
Encoding Source Language with Convolutional Neural Network for Machine Translation (2015, background): “Various models, mostly neural network-based, have been proposed for representing the source sentence, mainly as the encoder part in an encoder-decoder framework (Bengio et al., 2003; Auli et al., 2013; Kalch-brenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014).”
Hybrid Autoregressive Transducer (HAT) (2020, background): “Recent developments in the neural network literature, particularly sequence-to-sequence (Seq2Seq) modeling [12, 13, 14, 15, 16], have allowed the full optimization of the posterior probability, learning a direct mapping between feature sequences and orthographic-based (so-called character-based) lab”
Learning Semantic Representations for the Phrase Translation Model (2013, background): “, the feed-forward neural network language model (NNLM) [3] and the recurrent neural network language model (RNNLM) [19, 2], provide a different kind of latent semantic representation.”