Linguistic Input Features Improve Neural Machine Translation

By Rico Sennrich, Barry Haddow. 2016

400 citations recorded by the source corpus.

Neural machine translation has recently achieved impressive results, while using little in the way of external linguistic information. In this paper we show that the strong learning capability of neural MT models does not make linguistic features redundant; they can be easily incorporated to provide further improvements in performance. We generalize the embedding layer of the encoder in the attentional encoder--decoder architecture to support the inclusion of arbitrary features, in addition to the baseline word feature. We add morphological features, part-of-speech tags, and syntactic dependency labels as input features to English German, and English->Romanian neural machine translation systems. In experiments on WMT16 training and test sets, we find that linguistic input features improve model quality according to three metrics: perplexity, BLEU and CHRF3. An open-source implementation of our neural MT system is available, as are sample files and configurations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Contextual Sequence Modeling for Recommendation with Recurrent Neural Networks (2017, background): “Other works proposed including linguistic characteristics such as such as part of speech tags [18] or syntactic dependency information [14] about the current word.”
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation (2017, background): “Neural machine translation (NMT) is one of success stories of deep learning in natural language processing, with recent NMT systems outperforming traditional phrase-based approaches on many language pairs (Sennrich et al., 2016a).”
Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling (2017, background): “Beyond SRL, there have been many proposals on how to incorporate syntactic information in RNN models, for example, in the context of neural machine translation (Eriguchi et al., 2017; Sennrich and Haddow, 2016).”
Nematus: a Toolkit for Neural Machine Translation (2017, background): “• Instead of a single word embedding at each source position, our input representations allows multiple features (or “factors”) at each time step, with the final embedding being the concatenation of the embeddings of each feature (Sennrich and Haddow, 2016).”
The Lazy Encoder: A Fine-Grained Analysis of the Role of Morphology in Neural Machine Translation (2018, neutral): “Acquiring this knowledge is important to inform future research in NMT, especially regarding the usefulness of injecting linguistic information into the NMT model, e.g. by using supervised annotation (Sennrich and Haddow, 2016).”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, background): “Adding further source attributes, such as morphological segmentation labels, POS tags, and syntactic dependency labels, improves models, and concatenating or factorizing these with embeddings increases robustness further [248], [250].”
Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings (2017, background): “pute sense embeddings and include the resulting sense labels into the NMT model as additional input features (Alexandrescu and Kirchhoff, 2006; Sennrich and Haddow, 2016).”
Chunk-Based Bi-Scale Decoder for Neural Machine Translation (2017, methodology): “, 2016), or the sub-word level (Sennrich and Haddow, 2016; Sennrich et al., 2016; Garcı́a-Martı́nez et al., 2016) information, the corresponding NMT models capture the information inside the word and alleviate the problem of unknown words.”
Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models (2019, background): “This approach can also be seen as adding an additional token-level feature which is the same for all tokens in a sequence (Sennrich and Haddow, 2016).”
Neural Machine Translation with Source-Side Latent Graph Parsing (2017, background): “Note on character n -gram embeddings In NMT models, sub-word units are widely used to address rare or unknown word problems (Sennrich et al., 2016).”
Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation (2021, methodology): “Source-side divergence tags are integrated as feature factors (Had-dow and Koehn, 2012; Sennrich and Haddow, 2016; Hoang et al., 2016), while target-side divergence tags form an additional output sequence generated in a multi-task fashion (Garc´ıa-Mart´ınez et al., 2016, 2017).”
Knowledge Distillation for Bilingual Dictionary Induction (2017, methodology): “On the aspect of enriching word embeddings with linguistic knowledge for the purpose of machine translation, Sennrich and Barry (Sennrich and Haddow, 2016) introduce linguistic features in sequence to sequence neural machine translation.”
Learning to Parse and Translate Improves Neural Machine Translation (2017, background): “…demonstrated that NMT systems work comparably to other systems even when the source and target sentences are given simply as ﬂat sequences of characters (Lee et al., 2016; Chung et al., 2016) or statistically, not linguistically, motivated subword units (Sennrich et al., 2016; Wu et al., 2016).”
Improving the Robustness of Speech Translation (2018, methodology): “Due to this concern, we propose to apply a factored input embeddings by combining both character and Pinyin representations motivated by Sennrich and Haddow (2016).”
Automatically generating commit messages from diffs using neural machine translation (2017, methodology): “The default value of N is 4, which is used in our evaluation and is commonly used in other evaluations [26], [48], [34], [22], [33], [14].”
OpenNMT: Open-Source Toolkit for Neural Machine Translation (2017, methodology): “Case Study: Factored Neural Translation In feature-based factored neural translation (Sennrich and Haddow, 2016), instead of generating a word at each time step, the model generates both word and associated features.”
Augmenting Neural Machine Translation with Knowledge Graphs (2019, methodology): “Recent work has successfully devised strategies for incorporating different kinds of knowledge into NMT models, such as linguistic features (Sennrich and Haddow, 2016) and NE-tags (Gu et al.”
Ensembling Factored Neural Machine Translation Models for Automatic Post-Editing and Quality Estimation (2017, background): “tional word-level "factors" to source-word input embeddings (Sennrich and Haddow, 2016).”