Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models

By Minh-Thang Luong, Christopher D. Manning. 2016

381 citations recorded by the source corpus.

Nearly all previous work on neural machine translation (NMT) has used quite restricted vocabularies, perhaps with a subsequent method to patch in unknown words. This paper presents a novel word-character solution to achieving open vocabulary NMT. We build hybrid systems that translate mostly at the word level and consult the character components for rare words. Our character-level recurrent neural networks compute source word representations and recover unknown target words when needed. The twofold advantage of such a hybrid approach is that it is much faster and easier to train than character-based ones; at the same time, it never produces unknown words as in the case of word-based models. On the WMT'15 English to Czech translation task, this hybrid approach offers an addition boost of +2.1-11.4 BLEU points over models that already handle unknown words. Our best system achieves a new state-of-the-art result with 20.7 BLEU score. We demonstrate that our character models can successfully learn to not only generate well-formed words for Czech, a highly-inflected language with a very complex vocabulary, but also build correct representations for English source words.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massive Exploration of Neural Machine Translation Architectures (2017, neutral): “While some work (Luong et al., 2015b; Zhou et al., 2016; Luong and Manning, 2016; Wu et al., 2016) has achieved state of the art results using deep networks, others (Jean et al., 2015; Chung et al., 2016; Sennrich et al., 2016b) have achieved similar results with far shallower ones.”
Neural Machine Translation in Linear Time (2016, background): “The RNN Enc-Dec, however, does not preserve the source sequence resolution, a feature that ag-gravates learning for long sequences such as those that appear in character-to-character machine translation (Luong & Manning, 2016).”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “, chararacters [10], mixed word/characters [28], or more intelligent sub-words [38].”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Coverage issues arising from difﬁ-culty capturing all words of a language within a limited vocabulary budget promoted the development of character level systems (Ling et al., 2015; Luong and Manning, 2016; Chung et al., 2016; Costa-jussà and Fonollosa, 2016; Lee et al., 2017; Cherry et al., 2018).”
Enriching Word Vectors with Subword Information (2016, background): “Finally, recent works in machine translation have proposed using subword units to obtain representations of rare words (Sennrich et al., 2016; Lu-ong and Manning, 2016).”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, methodology): “ the language model to see this secret Split the numbers into each being their own word: “The random number is 1 2 3 4”. This approach is often taken in practice when using a limited-size vocabulary [33, 43]. Change the format of the secret to ﬁt the model, and use the English word for each number: ”
What do Neural Machine Translation Models Learn about Morphology? (2017, methodology): “…obtained in a pre-processing step—e.g. by byte-pair encoding (Sennrich et al., 2016) or the word-piece model (Wu et al., 2016)—or learned during training using a character-based convolutional/recurrent sub-network (Costa-jussà and Fonollosa, 2016; Luong and Manning, 2016; Vylomova et al., 2016).”
Zero-Resource Translation with Multi-Lingual Neural Machine Translation (2016, methodology): “Rather than word-level translation with language-speciﬁc preprocessing, neural machine translation has found to work well with statistically segmented subword sequences as well as sequences of characters (Chung et al., 2016; Luong and Manning, 2016; Sennrich et al., 2015b; Ling et al., 2015).”
How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs (2016, methodology): “We encourage the use of LingEval97 to assess alternative architectures, such as hybrid word-character models (Luong and Manning, 2016), or dilated convolutional networks (Kalchbrenner et al., 2016).”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, methodology): “Similarly, Ling et al. (2015) employed a bidirectional LSTM to compose character embeddings into word embeddings.”
Character-based Neural Machine Translation (2016, background): “Recently, (Luong and Manning, 2016) propose a combination of word and characters in neural MT.”
Universal Neural Machine Translation for Extremely Low Resource Languages (2018, neutral): “, 2016b) or characters (Kim et al., 2016; Luong and Manning, 2016; Lee et al., 2017).”
Charagram: Embedding Words and Sentences via Character n-grams (2016, methodology): “Luong et al. (2013) used recursive models to compose morphs into word”
Latent Sequence Decompositions (2016, background): “The output representation is usually a ﬁxed sequence of words (Sutskever et al., 2014; Cho et al., 2014), phonemes (Chorowski et al., 2015), characters (Chan et al., 2016; Bahdanau et al., 2016a) or even a mixture of characters and words (Luong & Manning, 2016).”
Compression of Neural Machine Translation Models via Pruning (2016, background): “;.'()*+#)+&'"#.-#.' Figure 1: A simpliﬁed diagram of NMT. Manning, 2015; Sennrich et al., 2016), EnglishTurkish (Sennrich et al., 2016), and English-Czech (Jean et al., 2015b; Luong and Manning, 2016). Figure 1 gives an example of an NMT system. While NMT has a signiﬁcantly s”
Words or Characters? Fine-grained Gating for Reading Comprehension (2016, methodology): “Character-level representations are also used to alleviate the difﬁculties of modeling out-of-vocabulary (OOV) tokens (Luong & Manning, 2016).”
Explaining Character-Aware Neural Networks for Word-Level Prediction: Do They Discover Linguistic Rules? (2018, positive): “, 2016) and machine translation (Luong and Manning, 2016).”
Finding Better Subword Segmentation for Neural Machine Translation (2018, methodology): “Luong and Manning [19] proposed a word-character hybrid model that translates mostly at the word level and consults the character information for rare words with an additional deep character-based LSTM.”