Deep Neural Networks as a Computational Model for Human Shape Sensitivity

By Jonas Kubilius, Stefania Bracci, Hans Op de Beeck. 2016

321 citations recorded by the source corpus.

Theories of object recognition agree that shape is of primordial importance, but there is no consensus about how shape might be represented, and so far attempts to implement a model of shape perception that would work with realistic stimuli have largely failed. Recent studies suggest that state-of-the-art convolutional ‘deep’ neural networks (DNNs) capture important aspects of human object perception. We hypothesized that these successes might be partially related to a human-like representation of object shape. Here we demonstrate that sensitivity for shape features, characteristic to human and primate vision, emerges in DNNs when trained for generic object recognition from natural photographs. We show that these models explain human shape judgments for several benchmark behavioral and neural stimulus sets on which earlier models mostly failed. In particular, although never explicitly trained for such stimuli, DNNs develop acute sensitivity to minute variations in shape and to non-accidental properties that have long been implicated to form the basis for object recognition. Even more strikingly, when tested with a challenging stimulus set in which shape and category membership are dissociated, the most complex model architectures capture human shape sensitivity as well as some aspects of the category structure that emerges from human judgments. As a whole, these results indicate that convolutional neural networks not only learn physically correct representations of object categories but also develop perceptually accurate representational spaces of shapes. An even more complete model of human object representations might be in sight by training deep architectures for multiple tasks, which is so characteristic in human development.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Geometry and Topology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Geometry and Topology

Citations in the corpus

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness (2018, background): “1 Moreover, CNNs have been proposed as computational models of human shape perception by Kubilius et al. (2016), who conducted an impressive number of experiments comparing human and CNN shape representations and concluded that CNNs “implicitly learn representations of shape that reﬂect human shape…”
Generalisation in humans and deep neural networks (2018, methodology): “For each of the twelve experiments that were conducted, either five or six observers participated (with the exception of the colour experiment, for which only three observers participated since similar experiments had already been performed by a number of studies [48, 55, 56]).”
The Notorious Difficulty of Comparing Human and Machine Perception (2020, background): “…(Geirhos, Temme, et al., 2018) or with respect to the type of features that play an important role for human or machine decision-making (Brendel & Bethge, 2019; Geirhos, Rubisch, et al., 2018; Kubilius, Bracci, & de Beeck, 2016; Ritter, Barrett, Santoro, & Botvinick, 2017; Ullman et al., 2016).”
Not-So-CLEVR: learning same–different relations strains feedforward neural networks (2018, background): “In addition, deep networks have been shown to be consistent with a number of human behaviours including rapid visual categorization [30,31], image memorability [32], typicality [33] as well as similarity [34] and shape sensitivity [35] judgements.”
Deep Neural Networks as Scientific Models. (2019, background): “A DNN trained on object categorisation predicted the perceived (similarity judgments) rather than the physical similarity (i.e., pixel values) of visual stimuli [10].”
Comparing deep neural networks against humans: object recognition when the signal gets weaker (2017, result): “Our ﬁrst experiment also shows that human observers’ accuracy suﬀers only marginally when images are converted to grayscale in comparison to coloured images, consistent with previous studies (Delorme et al., 2000; Kubilius, Bracci, & Op de Beeck, 2016; Wichmann, Braun, & Gegenfurtner, 2006) 8 .”
The Origins and Prevalence of Texture Bias in Convolutional Neural Networks (2019, background): “Previous studies found preferences for shape over color [77] and perceptual shape over physical shape [58].”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, background): “While an argument has been made for CNNs as a model of human shape sensitivity [40], other studies have demonstrated that CNNs rely too much on texture and not enough on shape when classifying images [41, 42].”
Towards deep learning with segregated dendrites (2016, background): “Interestingly, the representations that deep learning generates resemble those observed in the neocortex, particularly in higher-order sensory areas (Kubilius et al., 2016; Khaligh-Razavi and Kriegeskorte, 2014; Cadieu et al., 2014), suggesting that something akin to deep learning is occurring in…”
Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision (2016, background): “In line with previous studies (Khaligh-Razavi and Kriegeskorte 2014; Yamins et al. 2014; Güçlü and van Gerven 2015a, b; Cichy et al. 2016; Kubilius et al. 2016; Eickenberg et al. 2016; Horikawa and Kamitani 2017), we explored the correspondence between individual layers in the CNN and individual…”
Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks (2018, background): “…a large fraction of the variance in primate visual recognition behaviors (Rajalingham et al., 2015; Jozwik et al., 2016; Kheradpisheh et al., 2016; Kubilius et al., 2016; Peterson et al., 2016; Wallis et al., 2017), and the internal hidden neurons of those same models also predict a large…”
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs (2019, background): “Moreover, these networks also partly predict human and non-human primate object recognition performance and object similarity judgments [33, 19].”
Illusory Motion Reproduced by Deep Neural Networks Trained for Prediction (2018, methodology): “Several DNNs were used as computational models for human shape sensitivity; the output layers of the DNNs successfully developed representations that closely related human shape judgement (Kubilius et al., 2016).”