On Using Very Large Target Vocabulary for Neural Machine Translation

By Sébastien Jean, Kyunghyun Cho, Roland Memisevic, Yoshua Bengio. 2014

1,036 citations recorded by the source corpus.

Neural machine translation, a recently proposed approach to machine translation based purely on neural networks, has shown promising results compared to the existing approaches such as phrase-based statistical machine translation. Despite its recent success, neural machine translation has its limitation in handling a larger vocabulary, as training complexity as well as decoding complexity increase proportionally to the number of target words. In this paper, we propose a method based on importance sampling that allows us to use a very large target vocabulary without increasing training complexity. We show that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary. The models trained by the proposed approach are empirically found to outperform the baseline models with a small vocabulary as well as the LSTM-based neural machine translation models. Furthermore, when we use the ensemble of a few models with very large target vocabularies, we achieve the state-of-the-art translation performance (measured by BLEU) on the English!German translation and almost as high performance as state-of-the-art English!French translation system.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massive Exploration of Neural Machine Translation Architectures (2017, background): “Our model is outper-formed only by (Wu et al., 2016 (Jean et al., 2015), RNNSearch-LV (Jean et al., 2015), BPE (Sennrich et al., 2016b), BPE-Char (Chung et al., 2016), Deep-Att (Zhou et al., 2016), Luong (Lu-ong et al., 2015a), Deep-Conv (Gehring et al., 2016), GNMT (Wu et al., 2016), and OpenNMT…”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “Secondly, NMT lacks robustness in translating rare words.”
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, methodology): “, 2015) and RNNsearch-LV (Jean et al., 2015), achieve BLEU scores of 28.”
Multi-task Sequence to Sequence Learning (2015, background): “Recent work such as (Xu et al., 2015; Jean et al., 2015a; Luong et al., 2015a; Vinyals et al., 2015a) has found that it is crucial to empower seq2seq models with the attention mechanism.”
Neural Machine Translation of Rare Words with Subword Units (2015, methodology): “We generally follow settings by previous work (Bahdanau et al., 2015; Jean et al., 2015).”
Effective Approaches to Attention-based Neural Machine Translation (2015, methodology): “We carry out a similar set of experiments for the WMT’15 translation task from German to English.”
An efficient framework for learning sentence representations (2018, methodology): “This further limits the size of the vocabulary and the model (Variations of the softmax layer such as hierarchical softmax (Mnih & Hinton, 2009), sampling based softmax (Jean et al., 2014) and sub-word representations (Sennrich et al., 2015) can help alleviate this issue).”
Semi-supervised Sequence Learning (2015, methodology): “[32], which has been successfully used for machine translation [21, 11], text parsing [33], image captioning [35], video analysis [31], speech recognition [4] and conversational modeling [28, 34].”
A Call for Clarity in Reporting BLEU Scores (2018, methodology): “ich applied preprocessing to the reference, affecting scores by half a point. paper conﬁguration Chiang (2005) metric lc Bahdanau et al. (2014) (unclear) Luong et al. (2015b) user or metric (unclear) Jean et al. (2015) user Wu et al. (2016) user or user lc (unclear) Vaswani et al. (2017) user or use”
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (2016, methodology): “In this variant, we adapted the large vocabulary ‘trick’ (LVT) described in (Jean et al., 2014) to the summarization problem.”
TensorFlow: A system for large-scale machine learning (2016, neutral): “The sampled softmax (solid lines) reduces the data transferred and the computation performed at the PS tasks [35].”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Using PolyakRuppert averaging for model ensembling is a standard practice to improve the performance of a model [31].”
Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “The full softmax is approximated by 1024 samples as proposed in (Jean et al., 2015).”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “The sequence to sequence framework has been used extensively for many applications: machine translation [24, 25], image captioning [26, 27], parsing [28] and conversational modeling [29].”
Addressing the Rare Word Problem in Neural Machine Translation (2014, background): “, WMT’14 style, to be comparable with results reported on the WMT website4 and (b)tokenized translations , so as to be consistent with previous work (Cho et al., 2014; Bahdanau et al., 2015; Schwenk, 2014; Sutskever et al., 2014; Jean et al., 2015).”