Learning to Generalize: Meta-Learning for Domain Generalization

By Da Li, Yongxin Yang, Yi-Zhe Song, Timothy M. Hospedales. 2017

1,757 citations recorded by the source corpus.

Domain shift refers to the well known problem that a model trained in one source domain performs poorly when appliedto a target domain with different statistics. Domain Generalization (DG) techniques attempt to alleviate this issue by producing models which by design generalize well to novel testing domains. We propose a novel meta-learning method for domain generalization. Rather than designing a specific model that is robust to domain shift as in most previous DG work, we propose a model agnostic training procedure for DG. Our algorithm simulates train/test domain shift during training by synthesizing virtual testing domains within each mini-batch. The meta-optimization objective requires that steps to improve training domain performance should also improve testing domain performance. This meta-learning procedure trains models with good generalization ability to novel domains. We evaluate our method and achieve state of the art results on a recent cross-domain image classification benchmark, as well demonstrating its potential on two classic reinforcement learning tasks.

Subfields: Artificial Intelligence, Epidemiology, Microbiology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Epidemiology

In Microbiology

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Robust Global Representations by Penalizing Local Predictive Power (2019, methodology): “We test on the PACS data set (Li et al., 2017a), which consists of collections of images over four domains, including photo, art painting, cartoon, and sketch.”
Learning Robust Representations by Projecting Superficial Statistics Out (2018, methodology): “Finally, we tested on the PACS data set (Li et al., 2017a), which consists of collections of images of seven different objects over four domains, including photo, art painting, cartoon, and sketch.”
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (2017, methodology): “By incorporating prior knowledge about gradient-based learning, MAML improves on the statistical efﬁciency of black-box meta-learners and has successfully been applied to a range of meta-learning problems (Finn et al., 2017a;b; Li et al., 2017).”
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, background): “…from novel classes such that (i) the knowledge of base classes is not forgotten (preserved), as well as (ii) it quickly adapts to novel classes using few samples, (iii) thus, enabling it to perform well on real photos minimising the domain gap [26] with sketch samples from novel classes as support.”
Learngene: From Open-World to Your Learning Task (2021, background): “Although meta domain adaptation (Bal-aji, Sankaranarayanan, and Chellappa 2018; Li et al. 2018; Sun et al. 2019) uses a lightweight model, it cannot continually learn new claases in an open-world environment.”
SWAD: Domain Generalization by Seeking Flat Minima (2021, neutral): “…68.7 [31] 48.6 [32] 43.6 [15, 33] 65.3 SWAD (proposed) 88.1 79.1 70.6 50.0 46.5 66.9 Previous SOTA [31] + SWAD 88.3 78.9 71.3 51.0 46.8 67.3 12], obtained well-transferable model parameters by the meta-learning framework [13–16], data augmentation [17–19], or capturing causal relation [20, 21].”
Efficient Parametrization of Multi-domain Deep Neural Networks (2018, background): “[17] propose a meta-learning method that trains any given model to be more robust to domain shift.”
In Search of Lost Domain Generalization (2020, methodology): “Li et al. [2018a] employ Model-Agnostic Meta-Learning, or MAML [Finn et al., 2017], to build a predictor that learns how to adapt fast between training domains.”
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning (2018, background): “Many of these techniques aim to find a representation that is invariant to the domain [12, 16, 54, 41, 25].”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “Meta-learning can be an effective tool to support this goal by deﬁning the outer-loop validation set to have a domain shift with respect to the inner loop train set [58].”
Generalising Fine-Grained Sketch-Based Image Retrieval (2019, background): “Domain Generalisation The CC-FG-SBIR challenge can be seen as a special case of Domain Generalisation (DG) [16, 26, 18].”
Adversarial target-invariant representation learning for domain generalization (2019, background): “…has included the use data augmentation (Shankar et al., 2018; Volpi et al., 2018) at training time, meta-learning to simulate domain shift (Li et al., 2018a), adding a self-supervised task to encourage a feature extractor to learn better representations (Carlucci et al., 2019), or…”
Learning from Extrinsic and Intrinsic Supervisions for Domain Generalization (2020, methodology): “We followed the previous works in the literature [1,2,8,26] and employed the leave-one-domain-out cross-validation strategy to produce the experiment results, i.e., we take turns to choose each domain for testing, and train a network model with the remaining three domains.”
Learning to Generate Novel Domains for Domain Generalization (2020, positive): “This can be achieved by dividing source domains into meta-train and meta-test sets without overlapping, and training a model on the meta-train set such that the error on the meta-test set is reduced [26,2,10].”
FSDR: Frequency Space Domain Randomization for Domain Generalization (2021, methodology): “Most existing DG methods can be broadly categorized into multi-source DG [44, 13, 12, 34, 16, 33, 43, 42, 59, 36, 2, 10, 35] and single-source DG [64, 70, 77, 50] both of which strive to learn domainagnostic features by either learning a domain-invariant feature space [44, 13, 12, 16, 34, 59, 36, 10”
Domain Generalization Using a Mixture of Multiple Latent Domains (2019, methodology): “TF (Li et al. 2017a): The low-rank parameterized neural network, which reduces the number of parameters to be trained.”
SelfReg: Self-supervised Contrastive Regularization for Domain Generalization (2021, background): “Meta-learning frameworks [27] are also explored for domain generalization to metalearn how to generalize across domains by leveraging MAML [14].”
Domain-agnostic Question-Answering with Adversarial Training (2019, methodology): “We apply meta learning to domain generalization (Li et al., 2018a, 2019; Balaji et al., 2018) to simulate train/test domain shift.”