Character-based Neural Machine Translation

By Wang Ling, Isabel Trancoso, Chris Dyer, Alan W. Black. 2015

71 citations recorded by the source corpus.

We introduce a neural machine translation model that views the input and output sentences as sequences of characters rather than words. Since word-level information provides a crucial source of bias, our input model composes representations of character sequences into representations of words (as determined by whitespace boundaries), and then these are translated using a joint attention/translation model. In the target language, the translation is modeled as a sequence of word vectors, but each word is generated one character at a time, conditional on the previous character generations in each word. As the representation and generation of words is performed at the character level, our model is capable of interpreting and generating unseen word forms. A secondary benefit of this approach is that it alleviates much of the challenges associated with preprocessing/tokenization of the source and target languages. We show that our model can achieve translation results that are on par with conventional word-based models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Neural Machine Translation of Rare Words with Subword Units (2015, methodology): “Various techniques have been proposed to produce ﬁxed-length continuous word vectors based on characters or morphemes (Luong et al., 2013; Botha and Blunsom, 2014; Ling et al., 2015a; Kim et al., 2015).”
Character-based Neural Machine Translation (2016, background): “Recently, (Luong and Manning, 2016) propose a combination of word and characters in neural MT.”
Ensemble Learning for Multi-Source Neural Machine Translation (2016, methodology): “While this paper focuses on word-based models, the ensemble methods discussed in this paper can be applied to character-based sequential NMT models (Ling et al., 2015) in a very similar fashion.”
Finding Better Subword Segmentation for Neural Machine Translation (2018, background): “ networks to encode or decode the character sequences. For instance, Costa-jussa and Fonollosa [11] employed convolutional and highway layers upon characters to form the word embeddings. Ling et al. [18] used a bidirectional LSTM to combine character embeddings to word embeddings and generated the t”
Multi-task Prediction of Disease Onsets from Longitudinal Lab Tests (2016, methodology): “In natural language processing, for example, neural networks taking only character-level input achieve high performance on many tasks including text classiﬁcation tasks (Zhang et al., 2015; Kim, 2014), machine translation (Ling et al., 2015) and language modeling (Kim et al., 2016).”
How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse? (2017, background): “Examples for sub-word units include BPE based units (Sennrich et al., 2015), characters (Ling et al., 2015; Chung et al., 2016; Heigold et al., 2017) or even bytes (Gillick et al., 2015).”
Neural Lattice Language Models (2018, background): “kens x 1;x 2;:::;x jX such that X = [x 1;x 2;:::;x jX ], i.e. the concatenation of the tokens (with an appropriate delimiter). These tokens can be either on the character level (Hwang and Sung, 2017; Ling et al., 2015) or word level (Inan et al., 2017; Merity et al., 2016). Using the chain rule, lan”