Gradient Surgery for Multi-Task Learning

By Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn. 2020

1,977 citations recorded by the source corpus.

While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimization challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single-task learning are not fully understood. In this work, we identify a set of three conditions of the multi-task optimization landscape that cause detrimental gradient interference, and develop a simple yet general approach for avoiding such interference between task gradients. We propose a form of gradient surgery that projects a task's gradient onto the normal plane of the gradient of any other task that has a conflicting gradient. On a series of challenging multi-task supervised and multi-task RL problems, this approach leads to substantial gains in efficiency and performance. Further, it is model-agnostic and can be combined with previously-proposed multi-task architectures for enhanced performance.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “This is intuitive, given that multi-task learning causes gradients to be noisy (Yu et al., 2020).”
A Survey on Multi-Task Learning (2017, background): “Moreover, the GradNorm [115] is devised to normalize gradients to balance the learning of multiple tasks and [116] proposes the gradient surgery to avoid the interference between task gradients.”
Disentangling Transfer in Continual Reinforcement Learning (2022, methodology): “In this work, we focus on the SAC algorithm [16], which is often considered to be the method of choice for continuous control RL [48, 47, 37].”
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “In other words, the information carried by ∇L P and ∇L S might not mutually agree, and adding them naively would lead to inhibiting [69] the training signal overall.”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “Our work is related to (Yu et al. 2020) which considers interference between gradients in a multi-task setting.”
Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021, background): “some other learning paradigms also dealing with multiple tasks, such as multi-task learning (Yu et al., 2020) and transfer learning (Houlsby et al.”
A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix (2020, background): “The idea is to preserve the subspace spanned by the previous samples ([Yu et al., 2020] proposed a similar variant for multi-task learning ).”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “For instance, in metaworld domains, the standard Q-network used was [256, 256, 256] in size, and thus we used an error network of size: [256, 256, 256, 256], and for MT10 tasks we used [160, 160, 160, 160, 160, 160] sized Q-networks (Yu et al., 2020) and 1-extra layer error networks ∆φ.”
Meta-Learning in Neural Networks: A Survey (2020, background): “This may be partly due to conflicting gradients between tasks [289].”
ImageNet-21K Pretraining for the Masses (2021, methodology): “This extreme multi-task learning makes the optimization process harder, and may cause convergences to a weak local minimum [52, 5, 13].”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, background): “Recent work focuses on learning a single multi-task policy with both on-policy [34] and off-policy RL methods [35, 36, 37], and have shown promising results on a variety of robot manipulation tasks in simulation [34].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “To improve the multi-task performance, more advanced algorithms [141, 133] can be employed to mitigate the negative transfer effects.”
Organizing recurrent network dynamics by task-computation to enable continual learning (2020, methodology): “Previous approaches in this setting include regularisation of weight changes [13, 14], modiﬁcation to the learning signal [15–17], network architecture modiﬁcation [18, 19] or memory replay [20–22].”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, methodology): “Another set of methods seek to mitigate inter-task conflict by manipulating the direction of task gradients rather than simply their magnitude [48, 57, 53, 11, 50, 36].”