Character-based Neural Machine Translation

By Marta Ruiz Costa-jussà, José A. R. Fonollosa. 2016

345 citations recorded by the source corpus.

Neural Machine Translation (MT) has reached state-of-the-art results. However, one of the main challenges that neural MT still faces is dealing with very large vocabularies and morphologically rich languages. In this paper, we propose a neural MT system using character-based embeddings in combination with convolutional and highway layers to replace the standard lookup-based word representations. The resulting unlimited-vocabulary and affix-aware source word embeddings are tested in a state-of-the-art neural MT based on an attention-based bidirectional recurrent neural network. The proposed MT scheme provides improved results even when the source language is not morphologically rich. Improvements up to 3 BLEU points are obtained in the German-English WMT task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “Since then, many novel techniques have been proposed to further improve NMT: using an attention mechanism to deal with rare words [37], a mechanism to model translation coverage [42], multi-task and semi-supervised training to incorporate more data [14, 29], a character decoder [9], a character enco”
Sequence-Level Knowledge Distillation (2016, background): “…completely and obtaining word representations directly from characters with character composition models, which have many fewer parameters than word embedding lookup tables (Ling et al., 2015a; Kim et al., 2016; Ling et al., 2015b; Jozefowicz et al., 2016; Costa-Jussa and Fonollosa, 2016).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Coverage issues arising from difﬁ-culty capturing all words of a language within a limited vocabulary budget promoted the development of character level systems (Ling et al., 2015; Luong and Manning, 2016; Chung et al., 2016; Costa-jussà and Fonollosa, 2016; Lee et al., 2017; Cherry et al., 2018).”
What do Neural Machine Translation Models Learn about Morphology? (2017, methodology): “ing with a character-based convolutional/recurrent sub-network (Costa-jussà and Fonollosa, 2016; Luong and Manning, 2016; Vylomova et al., 2016).”
Zero-Resource Translation with Multi-Lingual Neural Machine Translation (2016, methodology): “Rather than word-level translation with language-speciﬁc preprocessing, neural machine translation has found to work well with statistically segmented subword sequences as well as sequences of characters (Chung et al., 2016; Luong and Manning, 2016; Sennrich et al., 2015b; Ling et al., 2015).”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “The most popular approach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutník et al., 2014; Bahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016).”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, background): “Costa-Juss´a and Fonollosa (2016) replaced the word-lookup table with convolutional and highway layers on top of character embeddings, while still segmenting source sentences into words.”
Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models (2016, background): “…line of work on end-to-end character-based neural models which achieve good results for part-of-speech tagging (dos Santos and Zadrozny, 2014; Ling et al., 2015a), dependency parsing (Ballesteros et al., 2015), text classiﬁcation (Zhang et al., 2015), speech recognition (Chan et al., 2016;…”
Multiplicative LSTM for sequence modelling (2016, background): “Character-level language modelling has also become of increased practical interest recently for use in machine translation (Ling et al., 2015), and in combination with word level models for general”
Investigating Language Universal and Specific Properties in Word Embeddings (2016, background): “Despite its popularity, character-based approach also comes out as an equal competitor (Santos and Zadrozny, 2014; Kim et al., 2016; Ling et al., 2015b; Ling et al., 2015a; Faruqui et al., 2016; Ballesteros et al., 2015) .”
Automatic Gender Identification and Reinflection in Arabic (2019, background): “The main reason for this setup is that character-level representations are reported to be good in capturing and learning morphological aspects (Ling et al., 2015; Kim et al., 2016), which is important for a morphologically rich language like Arabic.”
On Adversarial Examples for Character-Level Neural Machine Translation (2018, methodology): “The architecture we study was ﬁrst proposed by Kim et al. (2016) for language modeling, and later adapted by Costa-Jussa and Fonollosa (2016) for translation, and is also used by Belinkov and Bisk (2018) for their adversarially-trained models.”