ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

By Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, Wieland Brendel. 2018

3,222 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on "Stylized-ImageNet", a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “They distinguish these datasets, which all consist of novel images collected from a variety of sources, from synthetic distribution shifts such as ImageNet-C (Hendrycks & Dietterich, 2019), Stylized ImageNet (Geirhos et al., 2018), or adversarial attacks (Goodfellow et al.”
Shortcut learning in deep neural networks (2020, background): “Object textures and other local structures in images are highly useful for object classification in standard datasets [36], and DNNs strongly rely on texture cues for object classification, largely ignoring global object shape [37, 38].”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “CNNs have also been shown to over-rely upon texture rather than shape to make predictions, in contrast to human behavior [20].”
Self-Training With Noisy Student Improves ImageNet Classification (2019, neutral): “7 SIN [23] 224 45.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Recent work has connected lack of robustness to computer vision models’ encoding counterintuitive features (Ilyas et al., 2019; Geirhos et al., 2019; Yin et al., 2019; Wang et al., 2020).”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “Encouragingly, the past few years have seen substantial progress in robustness to these distribution shifts, e.g., see [13, 28, 34, 55, 57, 66, 93, 96, 105, 114, 115] among many others.”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, background): “, 2020 [14] find that many recent language models are already robust to many forms of distribution shift, while others [37, 10] find that vision models are largely fragile and argue that data augmentation offers one solution.”
Natural Adversarial Examples (2019, background): “, 2019 [17] propose making networks rely less on texture by training classifiers on images where textures are transferred from art pieces.”
Do Image Classifiers Generalize Across Time? (2019, background): “Recent work has proposed less adversarial image modifications such as small rotations & translations [6, 1, 7, 17], hue and color changes [14], image stylization [9] and synthetic image corruptions such as Gaussian blur and JPEG compression [13, 10].”
Learning Robust Global Representations by Penalizing Local Predictive Power (2019, methodology): “Also, notably, different from perturbed ImageNet validation sets (Geirhos et al., 2019; Hendrycks and Dietterich, 2019), the images of ImageNet-Sketch are collected independently from the original validation images.”
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “CutMix also generates new samples by cutting and pasting patches within mini-batches, leading to performance boosts in many computer vision tasks; unlike stylization as in [7], CutMix incurs only negligible additional cost for training.”
Analyzing and Improving the Image Quality of StyleGAN (2019, background): “Both FID and P&R are based on classifier networks that have recently been shown to focus on textures rather than shapes [10], and consequently, the metrics do not accurately capture all aspects of image quality.”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “As most neurons extract simple patterns and textures, the surprising effectiveness of training a network using a single image can be explained by the recent ﬁnding that even CNNs trained on ImageNet rely on texture (as opposed to shape) information to classify (Geirhos et al., 2019).”
Learning the Difference that Makes a Difference with Counterfactually-Augmented Data (2019, background): “For example, in computer vision, researchers have found that deep neural networks rely on surface-level texture [12, 5] or clues in the image’s background to recognize foreground objects even when that seems both unnecessary and somehow wrong: the beach is not what makes a seagull a seagull.”
Adversarial Filters of Dataset Biases (2020, background): “Large-scale neural networks have achieved superhuman performance across many popular AI benchmarks, for tasks as diverse as image recognition (ImageNet; Russakovsky et al., 1 Allen Institute for Artiﬁcial Intelligence 2 Paul G. Allen School of Computer Science, University of Washington.”