Learning Robust Representations by Projecting Superficial Statistics Out

By Haohan Wang, Zexue He, Zachary Chase Lipton, Eric P. Xing. 2018

253 citations recorded by the source corpus.

Despite impressive performance as evaluated on i.i.d. holdout data, deep neural networks depend heavily on superficial statistics of the training data and are liable to break under distribution shift. For example, subtle changes to the background or texture of an image can break a seemingly powerful classifier. Building on previous work on domain generalization, we hope to produce a classifier that will generalize to previously unseen domains, even when domain identifiers are not available during training. This setting is challenging because the model may extract many distribution-specific (superficial) signals together with distribution-agnostic (semantic) signals. To overcome this challenge, we incorporate the gray-level co-occurrence matrix (GLCM) to extract patterns that our prior knowledge suggests are superficial: they are sensitive to the texture but unable to capture the gestalt of an image. Then we introduce two techniques for improving our networks' out-of-sample performance. The first method is built on the reverse gradient method that pushes our model to learn representations from which the GLCM representation is not predictable. The second method is built on the independence introduced by projecting the model's representation onto the subspace orthogonal to GLCM representation's. We test our method on the battery of standard domain generalization data sets and, interestingly, achieve comparable or better performance as compared to other domain generalization methods that explicitly require samples from the target distribution for training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Robust Global Representations by Penalizing Local Predictive Power (2019, methodology): “As in Wang et al. (2019), training/validation samples are attached with patterns following two strategies: 1) independently : the pattern is independent of the digit, and 2) dependently : images of digit 0-4 have one pattern while images of digit 5-9 have the other pattern.”
Toward Causal Representation Learning (2021, background): “It also led to a range of follow-up work [32], [78], [97], [114], [115], [152], [153], [156], [167], [195], [204], [243], [263], [267], [277], [278], [281], complementing the studies of Bareinboim and Pearl [14], [185], and it inspired a thread of work in the statistics community exploiting…”
Causality for Machine Learning (2019, neutral): “…(Zhang et al., 2013; Weichwald et al., 2014; Zhang et al., 2015; Gong et al., 2016; Huang et al., 2017; Blöbaum et al., 2016; Guo et al., 2018; Wang et al., 2019; Subbaswamy et al., 2018; Rabanser et al., 2018; Li et al., 2018a; Lipton et al., 2018; Li et al., 2018b; Magliacane et al., 2018;…”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, background): “Another line of work has attempted to increase model robustness performance, either by directly projecting out superﬁcial statistics [48], via architectural improvements [5], pre-training schemes [25], or through the use of data augmentations.”
Learning De-biased Representations with Biased Representations (2019, background): “…X ) ⊥⊥ B | Y (called the “separation” constraint, Hardt et al. (2016)). to remove Other approaches have proposed predictability of p ( B | X ) based on f ( X ) through domain adversarial losses (Li & Vasconcelos, 2019; Wang et al., 2019b) or projection (Wang et al., 2019b; Quadrianto et al., 2019).”
Learning to Diversify for Single Domain Generalization (2021, methodology): “AGG [25], HEX [44], PAR [43], JiGen [5], ADA [42], MEADA [49], MMLD [31] and our method belong to the latter category.”
Towards Robustifying NLI Models Against Lexical Dataset Biases (2020, methodology): “…only as the evaluation set, we also add these samples back to the training set, similar to the concept of adversarial training (Jia and Liang, 2017; Wang et al., 2019c; Niu and Bansal, 2018) where the adversarial examples are added back to the training set so that the resulting model will be more…”
Self-Challenging Improves Cross-Domain Generalization (2020, methodology): “extracted robust feature representation by projecting out superficial patterns like color and texture [31].”
Out-of-Distribution Generalization via Risk Extrapolation (REx) (2020, background): “One direction in domain generalization is to explicitly project out superﬁcial domain-speciﬁc statistics (Wang et al., 2019) to reduce sensitivity to the domain.”
Domain Generalization: A Survey (2021, background): “Domain shift in reinforcement learning (RL) not only occurs in visual appearance (color/style changes, etc.) but also in other aspects like dynamics (transition function) or rewards (e.g., gravity/friction changes) [125].”