Vocabulary Manipulation for Neural Machine Translation

By Haitao Mi, Zhiguo Wang, Abe Ittycheriah. 2016

67 citations recorded by the source corpus.

In order to capture rich language phenomena, neural machine translation models have to use a large vocabulary size, which requires high computing time and large memory usage. In this paper, we alleviate this issue by introducing a sentence-level or batch-level vocabulary, which is only a very small sub-set of the full output vocabulary. For each sentence or batch, we only predict the target words in its sentence-level or batch-level vocabulary. Thus, we reduce both the computing time and the memory usage. Our method simply takes into account the translation options of each word or phrase in the source sentence, and picks a very small target vocabulary for each sentence based on a word-to-word translation model or a bilingual phrase library learned from a traditional machine translation model. Experimental results on the large-scale English-to-French task show that our method achieves better translation performance by 1 BLEU point over the large vocabulary neural machine translation system of Jean et al. (2015).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, methodology): “We use a target vocabulary of 160K words as well as vocabulary selection (Mi et al., 2016; L’Hostis et al., 2016) to decrease the size of the output layer which speeds up training and testing.”
A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “We use vocabulary selection which can speed up generation by up to a factor of ten at no cost in accuracy via making the time to compute the final output layer negligible (Mi et al., 2016; L’Hostis et al., 2016).”
Dynamic Language Models for Continuously Evolving Content (2021, background): “Simply designing a language model with overly large vocabularies cannot completely resolve the OOV issue, as new words are always emerging, while also being parametrically expensive [29, 37].”
Incorporating Discrete Translation Lexicons into Neural Machine Translation (2016, methodology): “ly have a more sophisticated method for choosing the interpolation between the standard and “copy” models. Incorporating these into our linearmethod is a promising avenue for future work. In addition Mi et al. (2016) have also recently proposed a similar approach by limiting the number of vocabulary”
Neural Machine Translation via Binary Code Prediction (2017, background): “Vocabulary selection approaches (Mi et al., 2016; L’Hostis et al., 2016) can also reduce the vocabulary size at testing, but these methods abandon full search over the target space and the quality of picked vocabularies directly affects the translation quality.”
Beam Search Strategies for Neural Machine Translation (2017, methodology): “During translation, we use the alignments (from the attention mechanism) to replace the unknown tokens either with potential targets (obtained from an IBM Model-1 trained on the parallel data) or with the source word itself (if no target was found) (Mi et al., 2016).”
Learning to Compute Word Embeddings On the Fly (2017, background): “This forces model designers to rely on overly large vocabularies, as observed by (Mi et al., 2016; Sennrich et al., 2015), which are parametrically expensive, or to employ vocabulary selection strategies (L’Hostis et al.”
Improving Lexical Choice in Neural Machine Translation (2017, background): “Some have focused on reducing the number of UNK s by enabling NMT to learn from a larger vocabulary (Jean et al., 2015; Mi et al., 2016); others have focused on replacing UNK s by copying source words (Gulcehre et al., 2016; Gu et al., 2016; Luong et al., 2015b).”
Coverage Embedding Models for Neural Machine Translation (2016, methodology): “Neural machine translation (NMT) has gained popularity in recent years (e.g. (Bahdanau et al., 2014; Jean et al., 2015; Luong et al., 2015; Mi et al., 2016b; Li et al., 2016)), especially for the attention-based models of Bahdanau et al. (2014).”
Neural Machine Translation: A Review (2019, unknown): “…Data filtering (Resnik, 1999; Khayrallah & Koehn, 2018; Carpuat et al., 2017; JunczysDowmunt, 2018a; Rossenbach et al., 2018; Junczys-Dowmunt, 2018b) • Word alignments (Mi et al., 2016b; Alkhouli & Ney, 2017; Alkhouli et al., 2016; Zenkel et al., 2019; Alkhouli et al., 2018; Stahlberg et al.,…”
Pinyin as Subword Unit for Chinese-Sourced Neural Machine Translation (2017, background): “With respect to the third category, extensive work has been done on using very large target vocabulary for NMT [6, 15, 10].”
On Learning Meaningful Code Changes Via Neural Machine Translation (2019, background): “This problem has been addressed by artiﬁcially limiting the vocabulary size, considering only most common words, assigning special tokens ( e.g., UNK ) to rare words or by learning subword units and splitting the words into constituent tokens [53], [67].”
Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions (2016, methodology): “Our implementation is based on [15], however, similar as [16], we find that relying mostly on K ′ does not result in deterioration of translation quality and that K can be greatly reduced.”