GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

By Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, Andrew Rabinovich. 2017

1,947 citations recorded by the source corpus.

Deep multitask networks, in which one neural network produces multiple predictive outputs, can offer better speed and performance than their single-task counterparts but are challenging to train properly. We present a gradient normalization (GradNorm) algorithm that automatically balances training in deep multitask models by dynamically tuning gradient magnitudes. We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting across multiple tasks when compared to single-task networks, static baselines, and other adaptive multitask loss balancing techniques. GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter $\alpha$. Thus, what was once a tedious search process that incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks. Ultimately, we will demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, neutral): “MTL requires the tasks to be trained from scratch every time, which makes it inefﬁcient and often requires careful weighting of the task-speciﬁc objective functions (Chen et al., 2017).”
Fine-tuned Language Models for Text Classification (2018, methodology): “MTL requires the tasks to be trained from scratch every time and often requires careful weighting of the task-specific objective functions (Chen et al., 2017).”
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021, background): “Various loss scaling techniques have been proposed, from dynamic scaling by inverse training loss to simple scaling by the number of data-points in respective datasets (Chen et al., 2018).”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “To suppress negative transfer, several approaches [6,21,59,16,69,56,39] dynamically re-weight each task’s loss function or re-order the task learning, to ﬁnd a ‘sweet spot’ where individual task performance does not degrade signiﬁcantly.”
ImageNet-21K Pretraining for the Masses (2021, methodology): “155 • Single-loss training - training with a softmax (a single loss) makes convergence easy and efficient, 156 and avoids many optimization problems associated with multi-loss learning, such as different 157 gradient magnitudes and gradient interference [59, 7, 9].”
How to Fine-Tune BERT for Text Classification? (2019, neutral): “MTL requires training tasks from scratch every time, which makes it inefﬁcient and it usually requires careful weighing of task-speciﬁc ob-jective functions (Chen et al., 2017).”
Learning to Fuse Things and Stuff (2018, background): “[7] propose another weighting based on gradient norms.”
Multi-Task Learning as Multi-Objective Optimization (2018, methodology): “The baselines we consider are (i) uniform scaling: minimizing a uniformly weighted sum of loss functions 1 T t L t , (ii) single task: solving tasks independently, (iii) grid search: exhaustively trying various values from { c t ∈ [0 , 1] | t c t = 1 } and optimizing for 1 T t c t L t , (iv) Kendall”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “GradNorm (Chen et al., 2017) is similarly inspired to these two methods, but doesn’t compute loss weights explicitly.”
Balancing Training for Multilingual Neural Machine Translation (2020, background): “Our work is also related to machine learning methods that balance multitask learning (Chen et al., 2018b; Kendall et al., 2018).”
Heterogeneous Few-Shot Model Rectification With Semantic Mapping (2020, background): “Thus how to balance different tasks [43], [44] is essential.”
NeurAll: Towards a Unified Visual Perception Model for Automated Driving (2019, background): “[37] achieved adaptive weighting by penalizing the network on higher magnitudes of back-propagated gradients.”
Temporal Attentive Alignment for Large-Scale Video Domain Adaptation (2019, methodology): “For future work, we plan to adopt adaptive weighting techniques used for multitask learning, such as uncertainty weighting [20] and GradNorm [4], to replace the manual grid-search method.”
Multi-task Learning (2020, methodology): “GradNorm was found to be much more sensitive to hyperparameters, and these were tuned via random search between [1e−6, 1e−2] for the learning rate and [1e−6, 5.0] for the spring constant.”