Fully Character-Level Neural Machine Translation without Explicit Segmentation

By Jason Lee, Kyunghyun Cho, Thomas Hofmann. 2016

469 citations recorded by the source corpus.

Most existing machine translation systems operate at the level of words, relying on explicit segmentation to extract tokens. We introduce a neural machine translation (NMT) model that maps a source character sequence to a target character sequence without any segmentation. We employ a character-level convolutional network with max-pooling at the encoder to reduce the length of source representation, allowing the model to be trained at a speed comparable to subword-level models while capturing local regularities. Our character-to-character model outperforms a recently proposed baseline with a subword-level encoder on WMT’15 DE-EN and CS-EN, and gives comparable performance on FI-EN and RU-EN. We then demonstrate that it is possible to share a single character-level encoder across multiple languages by training a model on a many-to-one translation task. In this multilingual setting, the character-level encoder significantly outperforms the subword-level encoder on all the language pairs. We observe that on CS-EN, FI-EN and RU-EN, the quality of the multilingual character-level translation even surpasses the models specifically trained on that language pair alone, both in terms of the BLEU score and human judgment.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “The proposed MetaNMT differs from the existing framework of multilingual translation (Lee et al., 2016; Johnson et al., 2016; Gu et al., 2018b) or transfer learning (Zoph et al., 2016).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “For the base-line experiment, we start by following common conventions in literature on multilingual NMT (Firat et al., 2017; Lee et al., 2017; Johnson et al., 2017).”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “…costs of machine learning models tend to scale with sequence length, much previous work on character- and bytelevel models has explored ways to process long sequences efficiently using convolutions with pooling (Zhang et al., 2015; Lee et al., 2017) or adaptive computation time (Graves, 2016).”
Complete Multilingual Neural Machine Translation (2020, neutral): “Sampling scheduling Several approaches proposed to address data sampling for multi-task models, some relying on temperature-based heuristics (Lee et al., 2016; Devlin et al., 2018; Arivazhagan et al., 2019b), others relying on adaptive schedules that incorporate the model gains, baselines or…”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “As has been shown in Lee et al. (2017), these models often overﬁt on low resource tasks and underﬁt on high resource tasks.”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, background): “Recently in [16] a CNN-based character-level encoder was proposed which is shared across multiple source languages.”
Unsupervised Neural Machine Translation (2017, methodology): “Our system makes use of one and only one encoder that is shared by both languages involved, similarly to Ha et al. (2016), Lee et al. (2017) and Johnson et al. (2017).”
Synthetic and Natural Noise Both Break Neural Machine Translation (2017, methodology): “See Lee et al. (2017) for architecture details.”
Natural Language Inference over Interaction Space (2017, methodology): “As in (Kim et al., 2016; Lee et al., 2016), The character feature is obtained by using a convolutional neural network followed by a max pooling over the learned character vectors.”
How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs (2016, methodology): “…text representations on the parallel part of the training set: • BPE-to-BPE (Sennrich et al., 2016a) • BPE-to-char (Chung et al., 2016) • char-to-char (Lee et al., 2016) We use the implementations released by the respective authors, Nematus6 for BPE-to-BPE, and dl4mt-c2c7 for BPE-to-char and…”
Tacotron: Towards End-to-End Speech Synthesis (2017, methodology): “CBHG is inspired from work in machine translation [8], where the main differences from [8] include using non-causal convolutions, batch normalization, residual connections, and stride=1 max pooling.”
Quasi-Recurrent Neural Networks (2016, methodology): “Convolutional models for sequence processing have been more successful when combined with RNN layers in a hybrid architecture (Lee et al., 2016), because traditional max-and average-pooling approaches to combining convolutional features across timesteps assume time invariance and hence cannot make…”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “CUNI’s (EN-DE) system is based on the character-to-character neural network architecture described in (Lee et al., 2016).”
Universal Neural Machine Translation for Extremely Low Resource Languages (2018, background): “Multi-lingual NMT is quite appealing for low-resource languages; several papers highlighted the characteristic that make it a good ﬁt for that such as Lee et al. (2017”
Training Tips for the Transformer Model (2018, methodology): “Lee et al., 2016). T2T adopts this vision and while it supports the use of external subword units, it comes with its own built-in method similar to the word-piece algorithm by Wu et al. (2016) and does not expect the input to be even tokenized.”
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation (2018, background): “Based on the literature, both requirements can be satisﬁed by training a multi-lingual NMT model having a shared encoder (Lee, Cho, and Hofmann 2017; Johnson et al. 2017), and a separate decoder and attention mechanism for each target language (Dong et al. 2015).”
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, neutral): “By contrast, Lee et al. (2017) exploited character-level inputs and adopted a shared encoder for many-to-one translation.”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “Our method outperforms CANINE-style downsampling (local attention, strided convolutions) and also leads to improvements in the monolingual setting, while using less compute and parameters to down-sample than both Lee et al. (2017) and Clark et al. (2021).”
Contextual Parameter Generation for Universal Neural Machine Translation (2018, methodology): “Lee et al. (2017) used a single character-level encoder across multiple languages by training a model on a many-to-one translation task.”