Deep convolutional networks do not classify based on global object shape

By Nicholas Baker, Hongjing Lu, Gennady Erlikhman, Philip J. Kellman. 2018

398 citations recorded by the source corpus.

Deep convolutional networks (DCNNs) are achieving previously unseen performance in object classification, raising questions about whether DCNNs operate similarly to human vision. In biological vision, shape is arguably the most important cue for recognition. We tested the role of shape information in DCNNs trained to recognize objects. In Experiment 1, we presented a trained DCNN with object silhouettes that preserved overall shape but were filled with surface texture taken from other objects. Shape cues appeared to play some role in the classification of artifacts, but little or none for animals. In Experiments 2–4, DCNNs showed no ability to classify glass figurines or outlines but correctly classified some silhouettes. Aspects of these results led us to hypothesize that DCNNs do not distinguish object’s bounding contours from other edges, and that DCNNs access some local shape features, but not global shape. In Experiment 5, we tested this hypothesis with displays that preserved local features but disrupted global shape, and vice versa. With disrupted global shape, which reduced human accuracy to 28%, DCNNs gave the same classification labels as with ordinary shapes. Conversely, local contour changes eliminated accurate DCNN classification but caused no difficulty for human observers. These results provide evidence that DCNNs have access to some local shape information in the form of local edge relations, but they have no access to global object shapes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Reducing Domain Gap by Reducing Style Bias (2021, background): “A recent line of studies has revealed that standard CNNs have an inductive bias far different from human vision: while humans tend to recognize objects based on their contents (i.e. shapes) [27], CNNs exhibit a strong bias towards styles (i.e. textures) [1, 14, 21].”
Semantic Pyramid for Image Generation (2020, methodology): “re of the castles has completely changed. However, local attributes such as the existence of small windows, towers remained (even though in different locations). This aligns with observations made by [17]. 4.1. Quantitative Evaluation We evaluated the quality of our generated images using two measur”
The Origins and Prevalence of Texture Bias in Convolutional Neural Networks (2019, background): “Our work builds on recent evidence for bias for texture versus shape in neural networks based on classiﬁcation behavior with ambiguous stimuli [ 36, 4].”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, methodology): “5 XSmall (XS) [1, 2, 4, 1] [ 64, 128, 192, 256] 10.”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, background): “While an argument has been made for CNNs as a model of human shape sensitivity [40], other studies have demonstrated that CNNs rely too much on texture and not enough on shape when classifying images [41, 42].”
Reducing Domain Gap via Style-Agnostic Networks (2019, neutral): “textures) [1,14,21].”