Comparing deep neural networks against humans: object recognition when the signal gets weaker

By Robert Geirhos, David H. J. Janssen, Heiko H. Schütt, Jonas Rauber, Matthias Bethge, Felix Wichmann. 2017

253 citations recorded by the source corpus.

Human visual object recognition is typically rapid and seemingly effortless, as well as largely independent of viewpoint and object orientation. Until very recently, animate visual systems were the only ones capable of this remarkable computational feat. This has changed with the rise of a class of computer vision algorithms called deep neural networks (DNNs) that achieve human-level classification performance on object recognition tasks. Furthermore, a growing number of studies report similarities in the way DNNs and the human visual system process objects, suggesting that current DNNs may be good models of human visual object recognition. Yet there clearly exist important architectural and processing differences between state-of-the-art DNNs and the primate visual system. The potential behavioural consequences of these differences are not well understood. We aim to address this issue by comparing human and DNN generalisation abilities towards image degradations. We find the human visual system to be more robust to image manipulations like contrast reduction, additive noise or novel eidolon-distortions. In addition, we find progressively diverging classification error-patterns between humans and DNNs when the signal gets weaker, indicating that there may still be marked differences in the way humans and current DNNs perform visual object recognition. We envision that our findings as well as our carefully measured and freely available behavioural datasets provide a new useful benchmark for the computer vision community to improve the robustness of DNNs and a motivation for neuroscientists to search for mechanisms in the brain that could facilitate this robustness.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “Geirhos et al. (2017) compare networks to humans on noisy and elastically deformed images.”
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness (2018, background): “Furthermore, while ImageNet-trained CNNs generalise poorly towards a wide range of image distortions (e.g. Dodge & Karam, 2017; Geirhos et al., 2017; 2018), our ResNet-50 trained on Stylized-ImageNet often reaches or even surpasses human-level robustness (without ever being trained on the specific i”
Generalisation in humans and deep neural networks (2018, background): “(2)We have reported a subset of these experiments on arXiv in an earlier version of this paper [41].”
Invariant object recognition is a personalized selection of invariant features in humans, not simply explained by hierarchical feed-forward vision models (2017, background): “Many previous studies have reported discrepancy between object recognition in humans and computational models of vision 31,56,57 .”
A parametric texture model based on deep convolutional features closely matches texture appearance for humans (2017, background): “Furthermore, human object recognition remains remarkably robust in images degraded by white noise, whereas the original VGG network is strongly impaired ( Geirhos et al., 2017).”
Human and DNN Classification Performance on Images With Quality Distortions (2019, background): “Deep networks can recognize objects in a scene (Girshick et al. 2014), classify every pixel in an image (Long et al. 2015), and hallucinate realistic-looking images (Karras et al. 2017), among many other applications. Recently, it has been reported that the performance of these deep networks exceeds”
Neural network models and deep learning - a primer for biologists (2019, background): “This has already helped reveal some limitations of convolutional neural networks, which appear to behave differently from humans under noisy conditions (Geirhos et al., 2017) and to show different patterns of failures across exemplars (Rajalingham et al.”
Adversarial Vision Challenge (2018, methodology): “The submitted Docker images were evaluated in the backend against the top-5 opponents (either models or attacks depending on track) on 200 validation samples to determine the score for that submission.”
Orthogonal Deep Neural Networks (2019, background): “Existing research suggests fragility of deep learning models to corruptions of such kinds [14], and that finetuning on specific corruption types would help, but cannot well generalize to other types of corruptions [16], [50].”
Totally Looks Like - How Humans Compare, Compared to Machines (2018, background): “Another line of work [17] compares the robustness of humans vs.”
PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition (2016, background): “undertook a similar study for image degradations [23].”