Shape or Texture: Understanding Discriminative Features in CNNs

By Md. Amirul Islam, Matthew Kowal, Patrick Esser, Sen Jia, Björn Ommer, Konstantinos G. Derpanis, Neil D. B. Bruce. 2021

93 citations recorded by the source corpus.

Contrasting the previous evidence that neurons in the later layers of a Convolutional Neural Network (CNN) respond to complex object shapes, recent studies have shown that CNNs actually exhibit a `texture bias': given an image with both texture and shape cues (e.g., a stylized image), a CNN is biased towards predicting the category corresponding to the texture. However, these previous studies conduct experiments on the final classification output of the network, and fail to robustly evaluate the bias contained (i) in the latent representations, and (ii) on a per-pixel level. In this paper, we design a series of experiments that overcome these issues. We do this with the goal of better understanding what type of shape information contained in the network is discriminative, where shape information is encoded, as well as when the network learns about object shape during training. We show that a network learns the majority of overall shape information at the first few epochs of training and that this information is largely encoded in the last few layers of a CNN. Finally, we show that the encoding of shape does not imply the encoding of localized per-pixel semantic information. The experimental results and findings provide a more accurate understanding of the behaviour of current CNNs, thus helping to inform future design choices.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs (2021, background): “We first validate how the top N overall position-encoding channels affect the performance [15] of a state-of-the-art semantic segmentation network, DeepLabv3-ResNet-50 [4], trained on the Cityscapes [7] dataset.”
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (2022, background): “Follow-up work explored this texture bias and showed that despite being texture-biased, CNNs still often represent information about the shape in their feature representations [37, 44].”
Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN (2023, methodology): “We follow the methodology in [80] to estimate the shape and texture information encoded in the latent representation.”
Intriguing Properties of Vision Transformers (2021, background): “Recently, [20] quantifies mutual information [21] between shape and texture features.”