Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models

By Zirui Wang, Yulia Tsvetkov, Orhan Firat, Yuan Cao. 2020

250 citations recorded by the source corpus.

Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization. While it is a common practice to apply a language-agnostic procedure optimizing a joint multilingual task objective, how to properly characterize and take advantage of its underlying problem structure for improving optimization efficiency remains under-explored. In this paper, we attempt to peek into the black-box of multilingual optimization through the lens of loss function geometry. We find that gradient similarity measured along the optimization trajectory is an important signal, which correlates well with not only language proximity but also the overall model performance. Such observation helps us to identify a critical limitation of existing gradient-based multi-task learning methods, and thus we derive a simple and scalable optimization procedure, named Gradient Vaccine, which encourages more geometrically aligned parameter updates for close tasks. Empirically, our method obtains significant model performance gains on multilingual machine translation and XTREME benchmark tasks for multilingual language models. Our work reveals the importance of properly measuring and utilizing language proximity in multilingual optimization, and has broader implications for multi-task learning beyond multilingual modeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Optimization & Training

Citations in the corpus

mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “interference by leveraging architectural or optimization improvements (Wang et al., 2020; Raffel et al., 2019).”
Scaling End-to-End Models for Large-Scale Multilingual ASR (2021, methodology): “In [27], a simple and scalable optimization procedure, namely Gradient Vaccine, is developed to address the gradient interference problem from different tasks.”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, background): “Another set of methods seek to mitigate inter-task conflict by manipulating the direction of task gradients rather than simply their magnitude [48, 57, 53, 11, 50, 36].”
Domain Generalization via Gradient Surgery (2021, methodology): “[32] generalize this idea by proposing an adaptive gradient similarity method (GradVac) that allows setting an individual gradient similarity objective for each task pair to better exploit inter-task correlations.”
An Empirical Study of Training End-to-End Vision-and-Language Transformers (2021, background): “performance drop is due to the conflicts between different objectives, and some techniques in multi-task optimization [49,53] may be borrowed to resolve the conflicts, which we list as one of the future directions.”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, background): “To address this issue, some previous works [58, 54, 8] try to modulate the gradient to prevent conﬂicts.”