Fast and Robust Neural Network Joint Models for Statistical Machine Translation

By Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard M. Schwartz, John I. Makhoul. 2014

590 citations recorded by the source corpus.

Recent work has shown success in using neural network language models (NNLMs) as features in MT systems. Here, we present a novel formulation for a neural network joint model (NNJM), which augments the NNLM with a source context window. Our model is purely lexicalized and can be integrated into any MT decoder. We also present several variations of the NNJM which provide significant additive improvements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “NMT is often accompanied by an attention mechanism [2] which helps it cope effectively with long input sequences.”
Sequence to Sequence Learning with Neural Networks (2014, methodology): “[8] followed a similar approach, but they incorporated their NNLM into the decoder of an MT system and used the decoder’s alignment information to provide the NNLM with the most useful words in the input sentence.”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, background): “More recently, Kalchbrenner and Blunsom (2013) and Devlin et al. (2014) reported the successful use of the neural networks as a sub-component of the existing translation system.”
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, neutral): “(Devlin et al., 2014) proposed to use a feedforward neural network to model a translation model, however, by predicting one word in a target phrase at a time.”
Semi-supervised sequence tagging with bidirectional language models (2017, background): “Recently, neural LMs (Bengio et al., 2003; Mikolov et al., 2010) have also been integrated in neural machine translation systems (e.g., Kalchbrenner and Blunsom, 2013; Devlin et al., 2014) to score candidate translations.”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, neutral): “Note that this model shares a resemblance with the NNJM of [41] for machine translation, where the previous word context are predicted words in the target language, and the forward context are words in the source language.”
Recurrent Neural Network Regularization (2014, methodology): “…recognition (Robinson et al., 1996; Graves et al., 2013) and have recently been applied to machine translation, where they are used for language modeling, re-ranking, or phrase modeling (Devlin et al., 2014; Kalchbrenner & Blunsom, 2013; Cho et al., 2014; Chow et al., 1987; Mikolov et al., 2013).”
Addressing the Rare Word Problem in Neural Machine Translation (2014, background): “In recent years, MT researchers have been seeking to incorporate neural network models into the standard pipeline as an additional subcomponent [20, 21, 23, 6].”
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses (2015, background): “Concatenating continuous representations prior to deep architectures is a common strategy to obtain order-sensitive representations (Bengio et al., 2003; Devlin et al., 2014).”
On Using Monolingual Corpora in Neural Machine Translation (2015, methodology): “Preprocessing Importantly, we did “not seg-ment” the Chinese sentences and considered each character as a symbol, unlike a more traditional approach of using a separate segmentation tool to segment the Chinese characters into words (Devlin et al. , 2014).”
Efficient softmax approximation for GPUs (2016, methodology): “Popular methods are Noise Contrastive Estimation [16, 37, 43] or a penalization on the normalization function [1, 10].”
Multi-Task Learning for Multiple Language Translation (2015, background): “Much work has been done to address the data sparsity problem such as the pivot language approach (Wu and Wang, 2007; Cohn and Lapata, 2007) and deep learning techniques (Devlin et al., 2014; Gao et al., 2014; Sundermeyer et al., 2014; Liu et al., 2014).”
Neural Text Generation from Structured Data with Application to the Biography Domain (2016, background): “, 2015), machine translation (Devlin et al., 2014; Bahdanau et al., 2015; Luong et al., 2015), and modeling conversations and dialogues (Shang et al.”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, background): “More recently, deep learning is making important strides in natural language processing, especially statistical machine translation [5, 6, 7].”
A Fast and Accurate Dependency Parser using Neural Networks (2014, methodology): “This is inspired by the recent success of distributed word representations in many NLP tasks, e.g., POS tagging (Collobert et al., 2011), machine translation (Devlin et al., 2014), and constituency parsing (Socher et al., 2013).”
Sequence-to-sequence neural net models for grapheme-to-phoneme conversion (2015, positive): “In recent work on sequence to sequence translation, it has been shown that side-conditioned neural networks can be effectively used for both machine translation [1–6] and image captioning [7–10].”
Binarized Neural Networks (2016, background): “ing but not limited to object recognition from images (Krizhevsky et al., 2012; Szegedy et al., 2014), speech recognition (Hinton et al., 2012; Sainath et al., 2013), statistical machine translation (Devlin et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015), Atari and Go games (Mnih et al.”
Attention with Intention for a Neural Network Conversation Model (2015, methodology): “In a neural network based machine translation system [6], the alignment information is used to outperform a strong phrase-based baseline [5].”