Addressing the Rare Word Problem in Neural Machine Translation

By Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, Wojciech Zaremba. 2014

807 citations recorded by the source corpus.

Neural Machine Translation (NMT) is a new approach to machine translation that has shown promising results that are comparable to traditional approaches. A significant weakness in conventional NMT systems is their inability to correctly translate very rare words: end-to-end NMTs tend to have relatively small vocabularies with a single unk symbol that represents every possible out-of-vocabulary (OOV) word. In this paper, we propose and implement an effective technique to address this problem. We train an NMT system on data that is augmented by the output of a word alignment algorithm, allowing the NMT system to emit, for each OOV word in the target sentence, the position of its corresponding word in the source sentence. This information is later utilized in a post-processing step that translates every OOV word using a dictionary. Our experiments on the WMT’14 English to French translation task show that this method provides a substantial improvement of up to 2.8 BLEU points over an equivalent NMT system that does not use this technique. With 37.5 BLEU points, our NMT system is the first to surpass the best result achieved on a WMT’14 contest task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “Our models are based on an encoder-decoder architecture with attention mechanism (Bahdanau et al., 2015; Luong et al., 2015a), as shown in ﬁg-ure 1.”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a).”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “Firstly, it generally takes a considerable amount of time and computational resources to train an NMT system on a large-scale translation dataset, thus slowing the rate of experimental turnaround time and innovation.”
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, methodology): “In encoding-decoding network, people used at most six LSTM layers (Luong et al., 2015).”
Exploring the Limits of Language Modeling (2016, methodology): “All experiments were run using the TensorFlow system (Abadi et al., 2015), with the exception of some older models which were used in the ensemble.”
Multi-task Sequence to Sequence Learning (2015, methodology): “The early seq2seq work (Sutskever et al., 2014; Cho et al., 2014; Luong et al., 2015b; Vinyals et al., 2015b) uses only the last encoder state to initialize the decoder and sets s = [ ] in Eq.”
Neural Machine Translation of Rare Words with Subword Units (2015, methodology): “For word-level NMT models, the translation of out-of-vocabulary words has been addressed through a back-off to a dictionary look-up (Jean et al., 2015; Luong et al., 2015b).”
Effective Approaches to Attention-based Neural Machine Translation (2015, methodology): “It is interesting to observe the trend previously reported in (Luong et al., 2015) that perplexity strongly correlates with translation quality.”
Grammar as a Foreign Language (2014, background): “It does, however, achieve high performance on a large scale machine tra slation task (Luong et al., 2014).”
Semi-supervised Sequence Learning (2015, methodology): “[32], which has been successfully used for machine translation [21, 11], text parsing [33], image captioning [35], video analysis [31], speech recognition [4] and conversational modeling [28, 34].”
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (2016, methodology): “, 2015) have also been used earlier for the problem of rare words in the context of machine translation (Luong et al., 2015), but the novel addition of switch in our model allows it to strike a balance between when to be faithful to the original source (e.”
Towards a Human-like Open-Domain Chatbot (2020, neutral): “Our work is therefore also related to past attempts to correlate perplexity with other automatic metrics in other tasks, e.g., perplexity vs. BLEU in translation (Luong et al., 2015).”
A Diversity-Promoting Objective Function for Neural Conversation Models (2015, background): “Recent progress in SMT stemming from the use of neural language models (Sutskever et al., 2014; Gao et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) has inspired attempts to extend these neural techniques to response generation.”
A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “The translation model computes a distribution over the V possible target words y i +1 by transforming the LSTM output h i via a linear layer with weights W o and bias b o : The conditional input c i at time i is computed via a simple dot-product style attention mechanism (Luong et al., 2015a).”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, neutral): “We evaluate our models using the standard BLEU score metric and to make our results comparable to previous work (Sutskever et al., 2014; Luong et al., 2015c; Zhou et al., 2016; Wu et al., 2016), we report tokenized BLEU score as computed by the multi-bleu.”
A Neural Attention Model for Abstractive Sentence Summarization (2015, methodology): “Similar issues have also been observed in neural translation models particularly in terms of translating rare words (Luong et al., 2015).”
A Neural Conversational Model (2015, methodology): “This framework has been used for neural machine translation and achieves improvements on the English-French and English-German translation tasks from the WMT’14 dataset (Luong et al., 2014; Jean et al., 2014).”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “The sequence to sequence framework has been used extensively for many applications: machine translation [24, 25], image captioning [26, 27], parsing [28] and conversational modeling [29].”
On Using Very Large Target Vocabulary for Neural Machine Translation (2014, methodology): “3 BLEU points away from the best system (Luong et al., 2015).”