Adaptive Scheduling for Multi-Task Learning

By Sébastien Jean, Orhan Firat, Melvin Johnson. 2019

50 citations recorded by the source corpus.

To train neural machine translation models simultaneously on multiple tasks (languages), it is common to sample each task uniformly or in proportion to dataset sizes. As these methods offer little control over performance trade-offs, we explore different task scheduling approaches. We first consider existing non-adaptive techniques, then move on to adaptive schedules that over-sample tasks with poorer results compared to their respective baseline. As explicit schedules can be inefficient, especially if one task is highly over-sampled, we also consider implicit schedules, learning to scale learning rates or gradients of individual tasks instead. These techniques allow training multilingual models that perform better for low-resource language pairs (tasks with small amount of data), while minimizing negative effects on high-resource tasks.

Subfields: Artificial Intelligence, Computer Networks and Communications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Orhan Firat
2. Ankur Bapna
3. Graham Neubig
4. Naveen Arivazhagan
5. Yulia Tsvetkov
6. Felix Stahlberg
7. Yonghui Wu
8. Melvin Johnson

In Computer Networks and Communications

1. Michael Crawshaw
2. Ankur Bapna
3. Yulia Tsvetkov
4. Anoop Kunchukuttan
5. Orhan Firat
6. Chenhui Chu
7. Raj Dabre

In Reinforcement Learning

1. Michael Crawshaw
2. Yulia Tsvetkov
3. Zirui Wang
4. Z. Chen
5. Yonghui Wu

Citations in the corpus

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “(Kiperwasser and Ballesteros, 2018) designed schedules favoring one target language and (Jean et al., 2018) learn an adaptive scheduler for multilingual NMT. Similar approaches could be extended in the context of learning different language pairs in massively multilingual settings.”
• Complete Multilingual Neural Machine Translation (2020, neutral): “…heuristics (Lee et al., 2016; Devlin et al., 2018; Arivazhagan et al., 2019b), others relying on adaptive schedules that incorporate the model gains, baselines or quality expectations into the data schedulers (Kiper-wasser and Ballesteros, 2018; Jean et al., 2019; Wang et al., 2020).”
• A Survey of Multilingual Neural Machine Translation (2019, background): “There are works that propose to focus on scaling learning rates or gradients differently for high-resource and low-resource language pairs [67].”
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “…training paradigm (Arivazhagan et al., 2019; Wang et al., 2020b), and how to further improve it by utilizing post-hoc alignment (Wang et al., 2020c; Cao et al., 2020), data balancing (Jean et al., 2019; Wang et al., 2020a), or calibrated training signal (Mulcaire et al., 2019; Huang et al., 2019a).”
• Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “To our knowledge, the only such works are Dynamic Task Prioritization (Guo et al., 2018) and the implicit schedule in (Jean et al., 2019).”