Decoding with Large-Scale Neural Language Models Improves Translation

By Ashish Vaswani, Yinggong Zhao, Victoria Fossum, David Chiang. 2013

276 citations recorded by the source corpus.

We explore the application of neural language models to machine translation. We develop a new model that combines the neural probabilistic language model of Bengio et al., rectified linear units, and noise-contrastive estimation, and we incorporate it into a machine translation system both by reranking k-best lists and by direct integration into the decoder. Our large-scale, large-vocabulary experiments across four language pairs show that our neural language model improves translation quality by up to 1.1 Bleu.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “The model was used to score partial translations during the decoding process, which generally leads to higher gains in BLEU score than n-best list rescoring (Vaswani et al., 2013).”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “, 2012) and statistical machine translation (Schwenk et al., 2012; Vaswani et al., 2013; Baltescu & Blunsom, 2015).”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “[51, 50] applied NCE to log-bilinear models and in [70] NCE is applied to neural probabilistic language models.”
Addressing the Rare Word Problem in Neural Machine Translation (2014, background): “In recent years, MT researchers have been seeking to incorporate neural network models into the standard pipeline as an additional subcomponent [20, 21, 23, 6].”
Efficient softmax approximation for GPUs (2016, methodology): “Noise Contrastive Estimation (Gutmann & Hyv ¨ arinen, 2010) replaces the softmax by a binary classiﬁer distinguishing the original distribution form a noisy one.”
Fast and Robust Neural Network Joint Models for Statistical Machine Translation (2014, methodology): “We should note that Vaswani et al. (2013) implements a method called Noise Contrastive Estimation (NCE) that is also used to train self-normalized NNLMs.”
Transfer Learning for Low-Resource Neural Machine Translation (2016, methodology): “It has a target vocabulary of 100K and is trained using noise-contrastive estimation (Mnih and Teh, 2012; Vaswani et al., 2013; Baltescu and Blunsom, 2014; Williams et al., 2015).”
Compositional Morphology for Word Representations and Language Modelling (2014, methodology): “Aside from the choice of language pairs, this evaluation diverges from Vaswani et al. (2013) by using normalised probabilities, a process made tractable by the class-based decomposition and caching of context-speciﬁc normaliser terms.”
A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “Sequence-level CRFs with neural-network clique potentials are discussed by Peng, Bo, and Xu (2009) and Do, Arti, and others (2010), where they are applied to sequence labeling of biological data, OCR data and speech signals, and by Wang and Manning (2013) who apply them on traditional natural langua”
Strategies for Training Large Vocabulary Neural Language Models (2015, background): “, 2012) and statistical machine translation (Schwenk et al., 2012; Vaswani et al., 2013; Baltescu and Blunsom, 2014).”
Auto-Sizing Neural Networks: With Applications to n-gram Language Models (2015, methodology): “The goal is to end up with a trained network words A Figure 1: Neural probabilistic language model (Bengio et al., 2003), adapted from Vaswani et al. (2013). that also has the optimal number of units in each layer.”
The Edinburgh/JHU Phrase-based Machine Translation Systems for WMT 2015 (2015, methodology): “For some language pairs (notably French ↔ English and Finnish ↔ English) we experimented with feed-forward neural network language models using the NPLM toolkit (Vaswani et al., 2013).”
Edinburgh’s Syntax-Based Systems at WMT 2015 (2014, methodology): “, 2014); a large-scale ngram Neural Network language model (NPLM) (Vaswani et al., 2013); treebank binarization (Sennrich and Haddow, 2015); particle verb restructuring (Sennrich and Haddow, 2015).”
Learning from past mistakes: improving automatic speech recognition output via noisy-clean phrase context modeling (2018, methodology): “ce, we adopt two types of neural network language models in this work. (i) Feed-forward neural network which is trained using a sequence of one-hot word representation along with the speciﬁed context [41]. (ii) Neural Dictionary Model Acoustic Model Language Model Lattice ASR Language Model (ARPA/Ne”
Alibaba Submission to the WMT18 Parallel Corpus Filtering Task (2018, background): “or NNJM, with 3 words of target context and 11 words of source context, effectively a 15-gram LM (Vaswani et al., 2013; Devlin et al., 2014).”
Variable length word encodings for neural translation models (2016, methodology): “The contribution of this dissertation is two-fold: • We implement a target side RNN language model based on Zaremba et al. (2014) using TensorFlow (Abadi et al.”
A Comparison of Neural Models for Word Ordering (2017, methodology): “N PLM , a 5-gram neural feedfor-ward language model, was trained for 10 epochs with a vocabulary size of 100k, 150 input and output units, 750 hidden units and 100 noise samples (Vaswani et al., 2013).”
Syntactically Guided Neural Machine Translation (2016, background): “Further gains can be achieved by adding a feed-forward neural language model with NPLM (Vaswani et al., 2013) (row 15).”
Improved Training Of Neural Trans-Dimensional Random field Language Models with Dynamic Noise-Contrastive Estimation (2018, methodology): “The NCE method is first proposed in [8], and has been used to train conditional neural network (NN) LMs, such as the feedforward NN LMs [12] and LSTM LMs [13], to avoid the expensive softmax calculation by treating the learning as a binary classification problem between the target words and the nois”