A Character-level Decoder without Explicit Segmentation for Neural Machine Translation

By Junyoung Chung, Kyunghyun Cho, Yoshua Bengio. 2016

339 citations recorded by the source corpus.

The existing machine translation systems, whether phrase-based or neural, have relied almost exclusively on word-level modelling with explicit segmentation. In this paper, we ask a fundamental question: can neural machine translation generate a character sequence without any explicit segmentation? To answer this question, we evaluate an attention-based encoder-decoder with a subword-level encoder and a character-level decoder on four language pairs--En-Cs, En-De, En-Ru and En-Fi-- using the parallel corpora from WMT'15. Our experiments show that the models with a character-level decoder outperform the ones with a subword-level decoder on all of the four language pairs. Furthermore, the ensembles of neural models with a character-level decoder outperform the state-of-the-art non-neural machine translation systems on En-Cs, En-De and En-Fi and perform comparably on En-Ru.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massive Exploration of Neural Machine Translation Architectures (2017, neutral): “Our model is outper-formed only by (Wu et al., 2016 (Jean et al., 2015), RNNSearch-LV (Jean et al., 2015), BPE (Sennrich et al., 2016b), BPE-Char (Chung et al., 2016), Deep-Att (Zhou et al., 2016), Luong (Lu-ong et al., 2015a), Deep-Conv (Gehring et al., 2016), GNMT (Wu et al., 2016), and OpenNMT (K”
Neural Machine Translation in Linear Time (2016, methodology): “…1.47 Layer-normalized LSTM (Chung et al., 2016a) 1.46 MI-LSTM (Wu et al., 2016b) 1.44 Recurrent Memory Array Structures (Rocki, 2016) 1.40 HM-LSTM (Chung et al., 2016a) 1.40 Layer Norm HyperLSTM (Ha et al., 2016) 1.38 Large Layer Norm HyperLSTM (Ha et al., 2016) 1.34 Recurrent Highway Networks…”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “Though this can be addressed in principle by training a “copy model” to mimic a traditional alignment model [31], or by using the attention mechanism to copy rare words [37], these approaches are both unreliable at scale, since the quality of the alignments varies across languages, and the latent…”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Coverage issues arising from difﬁ-culty capturing all words of a language within a limited vocabulary budget promoted the development of character level systems (Ling et al., 2015; Luong and Manning, 2016; Chung et al., 2016; Costa-jussà and Fonollosa, 2016; Lee et al., 2017; Cherry et al., 2018).”
A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “8 Chung et al. (2016) BPE-Char BiGRU Char 500 23.9 Yang et al. (2016) RNNSearch + UNK replace BiLSTM 50K 24.”
Filtering and Mining Parallel Data in a Joint Multilingual Space (2018, background): “…for two reasons: • it is the de-facto standard to evaluate NMT systems and many comparable results are available, e.g. (Sennrich et al., 2016b; Chunga et al., 2016; Wu et al., 2016; Gehring et al., 2017; Ashish Vaswani et al., 2017); • only a limited amount of parallel training data is…”
Zero-Resource Translation with Multi-Lingual Neural Machine Translation (2016, methodology): “More investigation with a diverse set of languages needs to be done in order to make a more solid conclusion, such as was done in (Firat et al., 2016; Chung et al., 2016).”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “The most popular approach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutník et al., 2014; Bahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016).”
How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs (2016, methodology): “…systems with different text representations on the parallel part of the training set: • BPE-to-BPE (Sennrich et al., 2016a) • BPE-to-char (Chung et al., 2016) • char-to-char (Lee et al., 2016) We use the implementations released by the respective authors, Nematus6 for BPE-to-BPE, and…”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, methodology): “Perhaps the work that is closest to our end goal is (Chung et al., 2016), which used a subword-level encoder from (Sennrich et al., 2015) and a fully character-level decoder ( bpe2char ).”
Character-based Neural Machine Translation (2016, background): “As further work, we are currently studying different alternatives (Chung et al., 2016) to extend the character-based approach to the target side of the neural MT system.”
Charagram: Embedding Words and Sentences via Character n-grams (2016, methodology): “Several have used character-level RNN architectures for machine translation, whether for representing source or target words (Ling et al., 2015b; Luong and Manning, 2016), or for generating entire translations character-by-character (Chung et al., 2016).”
SYSTRAN's Pure Neural Machine Translation Systems (2016, background): “A number of previous work such as character-level (Chung et al., 2016), hybrid word-character-based (Luong and Manning, 2016) and subword-level (Sennrich et al., 2016b) address issues that arise with morphologically rich languages such as German, Korean and Chinese.”
Neural Machine Translation: A Review (2019, methodology): “+ Character-level decoders better than subword-based ones in some studies [246, 269].”
A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions (2017, background): “…translation, neural MT (NMT), has emerged very recently and has already surpassed the performance of the mainstream approach in the ﬁeld, phrase-based MT (PBMT), for a number of language pairs, e.g. (Sennrich et al., 2015; Luong et al., 2015; Costa-Juss`a and Fonollosa, 2016; Chung et al., 2016).”
Finding Better Subword Segmentation for Neural Machine Translation (2018, methodology): “[9] introduced a character-level decoder without explicit segmentation while the encoder is still at subword-level.”
Neural Machine Translation with Byte-Level Subwords (2019, background): “(Kim et al. 2016) proposed building word representations from character ones; (Chung, Cho, and Bengio 2016) removed the re- striction of word boundaries and directly learned decoding in character level; (Lee, Cho, and Hofmann 2017) further extended it to a fully character-level model in a…”