Domain Generalization via Gradient Surgery

By Lucas Mansilla, Rodrigo Echeveste, Diego H. Milone, Enzo Ferrante. 2021

105 citations recorded by the source corpus.

In real-life applications, machine learning models often face scenarios where there is a change in data distribution between training and test domains. When the aim is to make predictions on distributions different from those seen at training, we incur in a domain generalization problem. Methods to address this issue learn a model using data from multiple source domains, and then apply this model to the unseen target domain. Our hypothesis is that when training with multiple domains, conflicting gradients within each mini-batch contain information specific to the individual domains which is irrelevant to the others, including the test domain. If left untouched, such disagreement may degrade generalization performance. In this work, we characterize the conflicting gradients emerging in domain shift scenarios and devise novel gradient agreement strategies based on gradient surgery to alleviate their effect. We validate our approach in image classification tasks with three multi-domain datasets, showing the value of the proposed agreement strategy in enhancing the generalization capability of deep learning models in domain shift scenarios.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, background): “Previous approaches in domain generalization are mainly based on gradient operations (Gulrajani and Lopez-Paz, 2020; Arjovsky et al., 2019; Mansilla et al., 2021; Kim et al., 2021) or disentanglement learning (Wang et al.”
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “Gradient Consensus: Inspired from multi-task learning [69] and domain generalisation [32] literature, we aim to update the model in the direction where there is an agreement in the gradient space between two domains in order to learn a domain invariant representation.”
Learning to Learn and Remember Super Long Multi-Domain Task Sequence (2022, unknown): “Gradient dot product information has been applied on various machine learning problem, including domain generalization [46, 64], multi-task learning [80] and continual learning [58].”