Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning

By Jan Niehues, Eunah Cho. 2017

119 citations recorded by the source corpus.

Linguistic resources such as part-of-speech (POS) tags have been extensively used in statistical machine translation (SMT) frameworks and have yielded better performances. However, usage of such linguistic annotations in neural machine translation (NMT) systems has been left under-explored. In this work, we show that multi-task learning is a successful and a easy approach to introduce an additional knowledge into an end-to-end neural attentional model. By jointly training several natural language processing (NLP) tasks in one system, we are able to leverage common information and improve the performance of the individual task. We analyze the impact of three design decisions in multi-task learning: the tasks used in training, the training schedule, and the degree of parameter sharing across the tasks, which is defined by the network architecture. The experiments are conducted for an German to English translation task. As additional linguistic resources, we exploit POS information and named-entities (NE). Experiments show that the translation quality can be improved by up to 1.5 BLEU points under the low-resource condition. The performance of the POS tagger is also improved using the multi-task learning scheme.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, background): “Moreover, we notice that in general higher BLEU score correspond to higher probing results, confirming the trend that encoding linguistic proprieties within the encoder representation go on par with better translation quality (Niehues and Cho, 2017; Kiperwasser and Ballesteros, 2018).”
Scheduled Multi-Task Learning: From Syntax to Translation (2018, methodology): “chosen sequence to sequence architecture, we study the impact of our method with a single (and shared across tasks) encoder and the architecture of separate decoders which has already proven to be a very effective multi-task learning scheme (Luong et al., 2015a; Niehues and Cho, 2017).”
Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation (2018, background): “Recently, (Niehues and Cho, 2017) have made use of several linguistic tasks to improve NMT.”
Natural Language Multitasking: Analyzing and Improving Syntactic Saliency of Hidden Representations (2018, background): “Niehues and Cho [2017] note that many linguistic resources that enabled SMT are not commonly used in NMT models.”
A Challenge Set and Methods for Noun-Verb Ambiguity (2018, methodology): “…tasks and explore if this dataset can be used directly in downstream tasks in a way similar to what have been done in Swayamdipta et al. (2017) and (Eriguchi et al., 2017; Niehues and Cho, 2017; Kiperwasser and Ballesteros, 2018) for injecting syntax in semantic role labeling and translation tasks.”
Improving Zero-shot Translation with Language-Independent Constraints (2019, background): “, 2016; Domhan and Hieber, 2017), it is not necessary to minimize the auto-encoder likelihood as in the multitask approach (Niehues and Cho, 2017), but only the decoder-level similarity between the true target sentence and the auto-encoded source sentence.”
AutoLoss: Learning Discrete Schedules for Alternate Optimization (2018, methodology): “Following Niehues and Cho (2017), we build an attention-based encoder-decoder model with three task objectives: the target task translates German into English ( (cid:96) 1 ), while the secondary tasks are German named entity recognition (NER) ( (cid:96) 2 ) and German POS tagging ( (cid:96) 3 ).”
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models (2019, unknown): “Mix-review can also be viewed as a form of multi-task learning (Li et al., 2019), which has been shown to be useful in neural machine translation (NMT) (Niehues & Cho, 2017), speech recognition (Toshniwal et al., 2017), optical character recognition (OCR) (Liao et al., 2019), etc.”
Multi-Task Learning for Sequence Tagging: An Empirical Study (2018, methodology): “Exceptions to this include the work of (Collobert et al., 2011), which considers four tasks: POS , CHUNK , NER , and SRL ; (Raganato et al., 2017), which considers three: word sense disambiguation with POS and coarse-grained semantic tagging based on WordNet lexicographer ﬁles; (Hashimoto et al., 20”