Learning Robust Global Representations by Penalizing Local Predictive Power

By Haohan Wang, Songwei Ge, Eric P. Xing, Zachary Chase Lipton. 2019

1,357 citations recorded by the source corpus.

Despite their renowned predictive power on i.i.d. data, convolutional neural networks are known to rely more on high-frequency patterns that humans deem superficial than on low-frequency patterns that agree better with intuitions about what constitutes category membership. This paper proposes a method for training robust convolutional networks by penalizing the predictive power of the local representations learned by earlier layers. Intuitively, our networks are forced to discard predictive signals such as color and texture that can be gleaned from local receptive fields and to rely instead on the global structures of the image. Across a battery of synthetic and benchmark domain adaptation tasks, our method confers improved generalization out of the domain. Also, to evaluate cross-domain transfer, we introduce ImageNet-Sketch, a new dataset consisting of sketch-like images, that matches the ImageNet classification validation set in categories and scale.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “They measure performance on a set of 7 distribution shifts: ImageNetV2 (Recht et al., 2019), ImageNet Sketch (Wang et al., 2019), Youtube-BB and ImageNet-Vid (Shankar et al., 2019), ObjectNet (Barbu et al., 2019), ImageNet Adversarial (Hendrycks et al., 2019), and ImageNet Rendition (Hendrycks et al”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Shifted performance evaluations form the backbone of empirical evaluations in the literature on robust machine learning and task adaptation (e.g., Hendrycks and Dietterich, 2019; Wang et al., 2019; Djolonga et al., 2020; Taori et al., 2020).”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, neutral): “[95], in more detail.”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, background): “ImageNet-R also includes the line drawings from [32], excluding horizontally mirrored duplicate images, pitch black images, and images from the incorrectly collected “pirate ship” class.”
Natural Adversarial Examples (2019, background): “A separate way to measure how well a network learns shape is to test generalization on paintings, cartoons, sketches [42], sculptures, toys, origami, embroidery, and so on, but we avoid such examples and only keep natural images during adversarial filtration.”
Towards Robust Vision Transformer (2021, methodology): “ImageNet-R [16] and ImageNet-Sketch [42] collect the online images consisting of naturally occurring distribution changes such as image style, to measure the generalization ability to new distributions at test time.”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “, 2021a), ImageNetSketch (Wang et al., 2019), ObjectNet (Barbu et al.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Robustness evaluations on benchmarks with natural distribution shifts (Hendrycks et al., 2021b,a; Recht et al., 2019; Barbu et al., 2019; Wang et al., 2019) show that the accuracy of models like CLIP and ALIGN typically drops less than 10%, while the accuracy of supervised and semi-supervised…”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “We evaluate robustness on ImageNetv2 [48], -R [26, 63], -A [27], -ReaL [2], and ObjectNet [1], following CLIP and ALIGN.”
Learning to Prompt for Vision-Language Models (2021, neutral): “, 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al.”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “We also evaluate ImageNet models on other distribution shift datasets, ImageNet-A [24], ImageNet-R [22], ImageNet-Sketch [61], ObjectNet [4], and ImageNet-C [23].”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “We follow [94] and evaluate robustness performance on ImageNet distribution shift datasets [3, 23, 25, 61, 82].”
Large image datasets: A pyrrhic win for computer vision? (2020, methodology): “Approaches include using hand-drawn sketch images (ImageNet-Sketch [82]), using GAN generated images [26] and techniques such as Dataset distillation [83],where a dataset or a subset of a dataset is distilled down to a few representative synthetic samples.”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, neutral): “7 Table 4: Zero-shot image classification results on ImageNet [9], ImageNet-A [64], ImageNet-R [65], ImageNet-V2 [66], ImageNet-Sketch [67] and ObjectNet [68].”
A ConvNet for the 2020s (2022, methodology): “We directly test our ImageNet-1K trained/fine-tuned classification models on several robustness benchmark datasets such as ImageNet-A [31], ImageNet-R [28], ImageNet-Sketch [73] and ImageNetC/C̄ [29, 45] datasets.”
Conditional Prompt Learning for Vision-Language Models (2022, methodology): “For domain generalization experiments, we use ImageNet as the source dataset and four other variants of ImageNet that contain different types of domain shift as the target datasets, namely ImageNetV2 [43], ImageNetSketch [50], ImageNet-A [22] and ImageNet-R [21].”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “We evaluate the PaLI checkpoints (without high-res phase) at 224×224 resolution on ImageNet and ImageNet OOD evaluation sets: ImageNet (Deng et al., 2009), ImageNet-R (Hendrycks et al., 2021a), ImageNet-A (Hendrycks et al., 2021b), ImageNet-Sketch (Wang et al., 2019b), ImageNet-v2 (Recht et al., 201”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Other similar benchmarks that have been proposed for domain generalization include VLCS (Fang et al., 2013), which tests generalization across similar visual object recognition datasets; PACS (Li et al., 2017), which (like DomainNet) tests generalization across datasets with different renditions; an”