Variable-Length Word Encodings for Neural Translation Models

By Rohan Chitnis, John DeNero. 2015

41 citations recorded by the source corpus.

Recent work in neural machine translation has shown promising performance, but the most eective architectures do not scale naturally to large vocabulary sizes. We propose and compare three variable-length encoding schemes that represent a large vocabulary corpus using a much smaller vocabulary with no loss in information. Common words are unaected by our encoding, but rare words are encoded using a sequence of two pseudo-words. Our method is simple and eective: it requires no complete dictionaries, learning procedures, increased training time, changes to the model, or new parameters. Compared to a baseline that replaces all rare words with an unknown word symbol, our best variable-length encoding strategy improves WMT English-French translation performance by up to 1.7 BLEU.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Neural Machine Translation of Rare Words with Subword Units (2015, background): “Our main claims are that the translation of rare and unknown words is poor in word-level NMT models, and that subword models improve the translation of these word types.”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, background): “A common approach for dealing with the open vocabulary issue is to break up rare words into subword units (Schuster and Nakajima, 2012; Chitnis and DeNero, 2015; Sennrich et al., 2016; Wu et al., 2016).”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, background): “…been shown to work well with word tokens (Bengio et al., 2001; Schwenk, 2007; Mikolov et al., 2010) but also with ﬁner units, such as subwords (Sen-nrich et al., 2015; Botha and Blunsom, 2014; Lu-ong et al., 2013) as well as symbols resulting from compression/encoding (Chitnis and DeNero, 2015).”
Neural Machine Translation: A Review (2019, methodology): “As compromise between characters and full words, compression methods like Huffman codes [251], word piece models [14, 252], or byte pair encoding [157, 253, BPE] can be”
BPE-Dropout: Simple and Effective Subword Regularization (2019, methodology): “Other segmentation algorithms include Creutz and Lagus (2006), Schuster and Nakajima (2012), Chitnis and DeNero (2015), Kunchukuttan and Bhattacharyya (2016), Wu and Zhao (2018), Banerjee and Bhattacharyya (2018).”
Variable length word encodings for neural translation models (2016, methodology): “A question arises from Chitnis and DeNero (2015), which is whether the individual symbols in the compact vocabulary can, in themselves, hold some semantic or syntactical utility in relation to the real words it represents. This leads to the question of whether we can break down rare words to smaller”
Syntactically Guided Neural Machine Translation (2016, background): “However, extending these representations to the large vocabularies needed for open-domain MT is an open area of research (Jean et al., 2015a; Luong et al., 2015; Sennrich et al., 2015; Chitnis and DeNero, 2015).”
Meaningless yet meaningful: Morphology grounded subword-level NMT (2018, methodology): “A word can be segmented as BPE (Sen-nrich et al., 2016), orthographic syllable (Kunchukuttan and Bhattacharyya, 2016), character (Chung et al., 2016; Costa-jussà and Fonollosa, 2016), Huffman encoding (Chitnis and DeNero, 2015).”
The Edit Distance Transducer in Action: The University of Cambridge English-German System at WMT16 (2016, methodology): “In the future, we are planning to add back-translation (Sennrich et al., 2016a) and investigate the use of character- or subword-based NMT (Sennrich et al., 2016b; Chitnis and DeNero, 2015; Ling et al., 2015; Chung et al., 2016; Luong and Manning, 2016) within our combination framework.”
Neural Machine Translation via Binary Code Prediction (2017, unknown): “Other methods using characters (Ling et al., 2015) or subwords (Sennrich et al., 2016; Chitnis and DeNero, 2015) can be applied to suppress the vocabulary size, but these methods also make for longer sequences, and thus are not a direct solution to problems of computational efﬁciency.”