Gradient Matching for Domain Generalization

By Yuge Shi, Jeffrey S. Seely, Philip H. S. Torr, Siddharth Narayanaswamy, Awni Y. Hannun, Nicolas Usunier, Gabriel Synnaeve. International Conference on Learning Representations, 2021

381 citations recorded by the source corpus.

Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their ability to generalize to unseen domains. Here, we propose an inter-domain gradient matching objective that targets domain generalization by maximizing the inner product between gradients from different domains. Since direct optimization of the gradient inner product can be computationally prohibitive -- requires computation of second-order derivatives -- we derive a simpler first-order algorithm named Fish that approximates its optimization. We demonstrate the efficacy of Fish on 6 datasets from the Wilds benchmark, which captures distribution shift across a diverse range of modalities. Our method produces competitive results on these datasets and surpasses all baselines on 4 of them. We perform experiments on both the Wilds benchmark, which captures distribution shift in the real world, as well as datasets in DomainBed benchmark that focuses more on synthetic-to-real transfer. Our method produces competitive results on both benchmarks, demonstrating its effectiveness across a wide range of domain generalization tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b7a4be5a703f251b2d341d30ccec5e201881981b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Qiao
2. Ziwei Liu
3. Chen Change Loy
4. Kaiyang Zhou
5. Tao A. Xiang
6. Li Shen
7. Tao Qin
8. Mingchen Gao

In Computer Vision and Pattern Recognition

1. Chen Change Loy
2. Yu Qiao
3. Luc Van Gool
4. Xiaogang Wang
5. Ziwei Liu
6. Tao A. Xiang
7. Dacheng Tao
8. Yi Yang

Citations in the corpus

• Prompt-aligned Gradient for Prompt Tuning (2022, background): “Another related field that leverages gradient matching to transfer knowledge is domain generalization [45, 38] and multi-task learning [44, 52].”
• Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, background): “[151] proposed a gradient-matching scheme, where their assumption is that the gradient direction of two domains should be the same to enhance”
• Domain Generalization: A Survey (2021, methodology): “Table 1 shows three datasets from WILDS that have been commonly used by the DG community [118], [119], [120].”
• Learning to Learn and Remember Super Long Multi-Domain Task Sequence (2022, positive): “Gradient dot product information has been applied on various machine learning problem, including domain generalization [46, 64], multi-task learning [80] and continual learning [58].”