Adapting Auxiliary Losses Using Gradient Similarity

By Yunshu Du, Wojciech M. Czarnecki, Siddhant M. Jayakumar, Razvan Pascanu, Balaji Lakshminarayanan. 2018

187 citations recorded by the source corpus.

One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations. However, it is not always trivial to know if an auxiliary task will be helpful for the main task and when it could start hurting. We propose to use the cosine similarity between gradients of tasks as an adaptive weight to detect when an auxiliary loss is helpful to the main loss. We show that our approach is guaranteed to converge to critical points of the main task and demonstrate the practical usefulness of the proposed algorithm in a few domains: multi-task supervised learning on subsets of ImageNet, reinforcement learning on gridworld, and reinforcement learning on Atari games.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning explanations that are hard to vary (2020, background): “Looking at agreement to characterize and improve the representations learned by neural networks is an idea that has been explored in recent work, mostly looking at the cosine similarity between gradients [Du et al., 2018, Eshratifar et al., 2018, Fort et al., 2019, Zhang et al., 2019b].”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, background): “Transfer matrix Generally, the relationship between tasks and its impact on learning dynamics of neural networks is hard to quantify, where semantic similarity does not typically lead to transfer [13].”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “Such methods can be bene ﬁ cial in other settings as well, for example, in multitask learning, where they have the potential to make learning more ef ﬁ - cient in the case of con ﬂ icting objectives [21 – 23].”
Gradient Surgery for Multi-Task Learning (2020, methodology): “Unlike Du et al. (2018), we use this measure for effective multi-task learning, instead of ignoring auxiliary objectives.”
ImageNet-21K Pretraining for the Masses (2021, background): “This extreme multi-task learning makes the optimization process harder, and may cause convergences to a weak local minimum [52, 5, 13].”
Meta-Curvature (2019, methodology): “This has been explicitly used in continual learning or multi-task learning setup to consider task similarity [7, 23, 36].”
Ray Interference: a Source of Plateaus in Deep Reinforcement Learning (2019, neutral): “Additionally, there are methods that prevent the interference directly at the gradient level [5, 47], normalize the scales of the losses [15], or explicitly preserve capacity for late-learned subtasks [18].”
A survey and critique of multiagent deep reinforcement learning (2018, background): “However, the idea of multiple predictions about arbitrary signals was originally suggested for RL, in the context of general value functions [315,317] and there still open problems, for example, better theoretical understanding [31,88].”
Multi-Task Reinforcement Learning with Soft Modularization (2020, methodology): “Researchers also propose to explicitly model the similarity between gradients from different tasks [44, 4, 15, 19, 35, 6, 42, 12].”
Semi-Supervised Learning by Label Gradient Alignment (2019, methodology): “…training points on a deep image classiﬁcation model (Koh & Liang, 2017), and in identifying helpful auxiliary tasks in reinforcement learning (Du et al., 2018), we propose to map points into model parameter space G using the model’s gradient: We use the resulting metric to impute labels for…”
Multi-Task Reinforcement Learning with Context-based Representations (2021, background): “Several works have focused on the problem of negative interference (Du et al., 2018; Suteu & Guo, 2019; Yu et al., 2020a) where the gradients corresponding to the different tasks interfere negatively with each other.”
Generalization in multitask deep neural classifiers: a statistical physics approach (2019, background): “The problem of increasing the range of parameters from which one gets a multitask benefit and decreasing potential harms has received increasing interest in recent years, often through clever loss or gradient weighting strategies [26, 27, 28].”
Adaptive Auxiliary Task Weighting for Reinforcement Learning (2019, methodology): “Cosine Similarity This baseline combines the gradients from the auxiliary tasks and the main task based on their cosine similarities [34].”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “…learning task relatedness (Zhang & Yeung, 2012), routing task-specifc paths (Rusu et al., 2016; Rosenbaum et al., 2019), altering gradients directly (Kendall et al., 2018; Chen et al., 2018a; Du et al., 2018; Yu et al., 2020), or searching pareto solutions (Sener & Koltun, 2018; Lin et al., 2019).”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “It should be noted that there are several related works (Xu et al., 2018b; Du et al., 2018) which introduce methods for weighting the loss of auxiliary tasks relative to a main task loss.”