Scheduled Multi-Task Learning: From Syntax to Translation

By Eliyahu Kiperwasser, Miguel Ballesteros. 2018

87 citations recorded by the source corpus.

Neural encoder-decoder models of machine translation have achieved impressive results, while learning linguistic knowledge of both the source and target languages in an implicit end-to-end manner. We propose a framework in which our model begins learning syntax and translation interleaved, gradually putting more focus on translation. Using this approach, we achieve considerable improvements in terms of BLEU score on relatively large parallel corpus (WMT14 English to German) and a low-resource (WIT German to English) setup.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “(Kiperwasser and Ballesteros, 2018) designed schedules favoring one target language and (Jean et al., 2018) learn an adaptive scheduler for multilingual NMT. Similar approaches could be extended in the context of learning different language pairs in massively multilingual settings.”
Complete Multilingual Neural Machine Translation (2020, neutral): “, 2019b), others relying on adaptive schedules that incorporate the model gains, baselines or quality expectations into the data schedulers (Kiperwasser and Ballesteros, 2018; Jean et al., 2019; Wang et al., 2020).”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, neutral): “Moreover, we notice that in general higher BLEU score correspond to higher probing results, confirming the trend that encoding linguistic proprieties within the encoder representation go on par with better translation quality (Niehues and Cho, 2017; Kiperwasser and Ballesteros, 2018).”
Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets (2019, background): “Better scheduling of multi-task training as presented by Kiperwasser and Ballesteros (2018) may also reduce the performance loss in such scenarios.”
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, methodology): “Specifically, to better balance the multi-task learning and fine-tuning, λ(t) is calculated as the sigmoid annealing function (Bowman et al., 2016; Kiperwasser and Ballesteros, 2018):”
Multi-Task Learning for Sequence Tagging: An Empirical Study (2018, background): “…lexicographer ﬁles; (Hashimoto et al., 2017), which considers ﬁve: POS , CHUNK , DEP , semantic relatedness, and textual entailment; (Niehues and Cho, 2017; Kiperwasser and Ballesteros, 2018), which both consider three: TRANSLATE with POS and NER , and TRANSLATE with POS and DEP , respectively.”
Ultra Fast Speech Separation Model with Teacher Student Learning (2021, methodology): “To avoid this limitation, we introduce the Objective Shifting mechanism to train the student with both the teacher’s prediction and training datasets [17, 18].”
Multilingual Neural Machine Translation with Task-Specific Attention (2018, methodology): “Our attention model variants can also be applied to multi-task learning frameworks, e.g. to improve the quality of translation using knowledge learned from a variety of other natural language processing tasks such as POS tagging and dependency parsing (Kiperwasser and Ballesteros, 2018).”
An Empirical Exploration of Curriculum Learning for Neural Machine Translation (2018, background): “Curriculum design is also a concern when deciding how to schedule learning from samples of different tasks either in a sequence from simpler to more difﬁcult tasks (Collobert and Weston, 2008) or in a multi-task learning framework (Graves et al., 2017; Kiperwasser and Ballesteros, 2018).”
A Challenge Set and Methods for Noun-Verb Ambiguity (2018, methodology): “…tasks and explore if this dataset can be used directly in downstream tasks in a way similar to what have been done in Swayamdipta et al. (2017) and (Eriguchi et al., 2017; Niehues and Cho, 2017; Kiperwasser and Ballesteros, 2018) for injecting syntax in semantic role labeling and translation tasks.”
Deep Contextualized Self-training for Low Resource Dependency Parsing (2019, background): “(AD-PDH): The absolute difference between the positional distance of a word from its head word according to the gold tree and the corresponding number according to the predicted tree (Kiperwasser and Ballesteros, 2018) (we count the words that separate the head from the modifier in the sentence, con”
Adaptive Scheduling for Multi-Task Learning (2019, background): “Kipperwasser and Ballesteros [10] also propose variable schedules that increasingly favor some tasks over time.”
Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (2018, background): “Kiperwasser and Ballesteros (2018) investigate how to interleave learning syntax and translation and how to schedule these tasks.”