Generalizing Across Domains via Cross-Gradient Training

By Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, Sunita Sarawagi. 2018

586 citations recorded by the source corpus.

We present CROSSGRAD, a method to use multi-domain training data to learn a classifier that generalizes to new domains. CROSSGRAD does not need an adaptation phase via labeled or unlabeled data, or domain features in the new domain. Most existing domain adaptation methods attempt to erase domain signals using techniques like domain adversarial training. In contrast, CROSSGRAD is free to use domain signals for predicting labels, if it can prevent overfitting on training domains. We conceptualize the task in a Bayesian setting, in which a sampling step is implemented as data augmentation, based on domain-guided perturbations of input instances. CROSSGRAD parallelly trains a label and a domain classifier on examples perturbed by loss gradients of each other's objectives. This enables us to directly perturb inputs, without separating and re-mixing domain signals while making various distributional assumptions. Empirical evaluation on three different applications where this setting is natural establishes that (1) domain-guided perturbation provides consistently better generalization to unseen domains, compared to generic instance perturbation methods, and that (2) data augmentation is a more stable and accurate method than domain adversarial training.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Toward Domain-Invariant Speech Recognition via Large Scale Training (2018, methodology): “We also expect modeling strategies like domain adversarial training [18, 37, 38] and better training techniques [39] to improve performance.”
Learning Robust Representations by Projecting Superficial Statistics Out (2018, methodology): “We compared the performance of HEX/ADV with several methods tested on this data including CAE (Rifai et al., 2011), MTAE (Ghifary et al., 2015), CCSA (Motiian et al., 2017), DANN (Ganin et al., 2016), Fusion (Mancini et al., 2018), LabelGrad, and CrossGrad (Shankar et al., 2018).”
Generalizing to Unseen Domains via Adversarial Data Augmentation (2018, background): “Domain generalization methods [28, 22, 27, 33, 24] that propose different ways to better generalize to unknown domains are also related to our work.”
Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation (2020, background): “Another stream of work learns to augment the input data with adversarial learning (Shankar et al., 2018; Volpi et al., 2018).”
Improving Confidence Estimates for Unfamiliar Examples (2018, background): “Domain generalization [32, 26, 45] aims to build models that generalize well on a previously unspeciﬁed domain, whose distribution can be different from all training domains.”
Domain Generalization with MixStyle (2021, methodology): “…CCSA (Motiian et al., 2017) and MMD-AAE (Li et al., 2018b), Jigsaw puzzle-based JiGen (Carlucci et al., 2019), adversarial gradient-based CrossGrad (Shankar et al., 2018), meta-learning-based Metareg (Balaji et al., 2018) and Epi-FCR (Li et al., 2019), and data augmentation-based L2A-OT (Zhou et…”
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts (2019, background): “…multiple environment distributions, some of which are available during training (source), while others are used for testing (target) (Li et al., 2018; Shankar et al., 2018; Muandet et al., 2013; Balaji et al., 2018; Ghifary et al., 2015; Motiian et al., 2017; Li et al., 2017a; Gan et al., 2016).”
Learning to Generate Novel Domains for Domain Generalization (2020, unknown): “These include (1) CrossGrad [47], the most related work that perturbs input using adversarial gradients from a domain classifier; (2) CCSA [36], which”
Style Normalization and Restitution for Generalizable Person Re-Identification (2020, methodology): “CrossGrad [35] generates pseudo training instances by pertubations in the loss gradients of the domain classifier and category classifier respectively.”
Learning to Diversify for Single Domain Generalization (2021, background): “[39] proposed to augment the source samples along the direction of the domain change.”
FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space (2021, methodology): “There are other methods tackling DG by manipulating deep neural network architectures [19, 23, 35], leveraging self-supervision signals [3, 54], designing training heuristics [17, 25], or conducting data augmentations [48, 53, 60, 61], which are free from requirement of data centralization.”
Domain Generalization using Causal Matching (2020, methodology): “The results for the prior approaches CCSA (Motiian et al., 2017), D-MTAE (Ghifary et al., 2015), LabelGrad (Goodfellow et al., 2014), DAN (Ganin et al., 2016), and CrossGrad (Shankar et al., 2018) are taken from Table 9 in (Shankar et al., 2018).”
Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, methodology): “[Shankar et al., 2018] used a Bayesian network to model dependence between label, domain and input instance, and proposed CrossGrad, a cautious data augmentation strategy that perturbs the input along the direction of greatest domain change while changing the class label as little as possible.”
Domain Generalization via Model-Agnostic Learning of Semantic Features (2019, background): “Data augmentation strategies, such as gradient-based domain perturbation [47] or adversarially perturbed samples [53] demonstrate effectiveness for model generalization.”
Heterogeneous Domain Generalization Via Domain Mixup (2020, methodology): “PT CrossGrad [10] MR [11] MR-FL [14] Reptile [12] AGG [14] FC [14] MIXUP MIX-ALL Aircraft 11.”
Episodic Training for Domain Generalization (2019, methodology): “As a result, our time cost is on par with MetaReg [1] and faster than MLDG [18] and CrossGrad [32].”
Domain Generalization by Solving Jigsaw Puzzles (2019, methodology): “Existing methods range from decoupling image style from the shared object content [3], to pulling data of different domains together and imposing adversarial conditions [27, 28], up to generating new samples to better cover the space spanned by any future target [39, 46].”