Vocabulary Selection Strategies for Neural Machine Translation

By Gurvan L'Hostis, David Grangier, Michael Auli. 2016

43 citations recorded by the source corpus.

Classical translation models constrain the space of possible outputs by selecting a subset of translation rules based on the input sentence. Recent work on improving the efficiency of neural translation models adopted a similar strategy by restricting the output vocabulary to a subset of likely candidates given the source. In this paper we experiment with context and embedding-based selection methods and extend previous work by examining speed and accuracy trade-offs in more detail. We show that decoding time on CPUs can be reduced by up to 90% and training time by 25% on the WMT15 English-German and WMT16 English-Romanian tasks at the same or only negligible change in accuracy. This brings the time to decode with a state of the art neural translation system to just over 140 words per seconds on a single CPU core for English-German.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, methodology): “We use a target vocabulary of 160K words as well as vocabulary selection (Mi et al., 2016; L’Hostis et al., 2016) to decrease the size of the output layer which speeds up training and testing.”
A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “We use vocabulary selection which can speed up generation by up to a factor of ten at no cost in accuracy via making the time to compute the final output layer negligible (Mi et al., 2016; L’Hostis et al., 2016).”
An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation (2016, background): “In a recent paper (L’Hostis et al., 2016), the authors show that CPU decoding time can be reduced by 90% by reducing the computation of the output layer, resulting in just over 140ms per sentence.”
Neural Machine Translation: A Review (2019, unknown): “, 2017) and embedding-based (L’Hostis et al., 2016) methods.”
Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog (2018, methodology): “mined by word alignments as well as the 2,000 most frequent words (L’Hostis et al., 2016).”
Pinyin as Subword Unit for Chinese-Sourced Neural Machine Translation (2017, background): “Recent work has been done in improving the generalisation capability of NMT for open vocabulary [14, 6, 11, 12, 19, 10].”
Learning to Compute Word Embeddings On the Fly (2017, background): “, 2015), which are parametrically expensive, or to employ vocabulary selection strategies (L’Hostis et al., 2016).”