Why is Real-World Visual Object Recognition Hard?

By Nicolas Pinto, David D. Cox, James J. DiCarlo. 2008

623 citations recorded by the source corpus.

Progress in understanding the brain mechanisms underlying vision requires the construction of computational models that not only emulate the brain's anatomy and physiology, but ultimately match its performance on visual tasks. In recent years, “natural” images have become popular in the study of vision and have been used to show apparently impressive progress in building such models. Here, we challenge the use of uncontrolled “natural” images in guiding that progress. In particular, we show that a simple V1-like model—a neuroscientist's “null” model, which should perform poorly at real-world visual object recognition tasks—outperforms state-of-the-art object recognition systems (biologically inspired and otherwise) on a standard, ostensibly natural image recognition test. As a counterpoint, we designed a “simpler” recognition test to better span the real-world variation in object pose, position, and scale, and we show that this test correctly exposes the inadequacy of the V1-like model. Taken together, these results demonstrate that tests based on uncontrolled natural images can be seriously misleading, potentially guiding progress in the wrong direction. Instead, we reexamine what it means for images to be natural and argue for a renewed focus on the core problem of object recognition—real-world image variation.

Subfields: Biophysics, Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation (2015, methodology): “For structures with a higher degree of non-linearity, such as local renormalization [58, 59], Taylor approximation applied to neuron activation xj can be used again to achieve an approximation for the structure as given in Eq (63).”
Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields (2010, background): “There are several system of this class (Lazebnik et al., 2006; Pinto et al., 2008, 2009; Serre et al., 2007; Yang et al., 2009; Lee et al., 2009; Yu et al., 2009) but speciﬁcally convolutional nets achieve 67% (Jarrett et al., 2009).”
Learning Human Pose Estimation Features with Convolutional Networks (2013, methodology): “The convnet pipeline shown in Figure 2 starts with a 64 × 64 pixel RGB input patch which has been local contrast normalized (LCN) [22] to emphasize geometric discontinuities and improve generalization performance [32].”
A High-Throughput Screening Approach to Discovering Good Forms of Biologically Inspired Visual Representation (2009, positive): “This recognition test has the benefit of being relatively quick to evaluate (because it only contains two classes), while at the same time having previous empirical grounding as a challenging object recognition test due to the large amount of position, scale, view, and background variation [11] (see”
Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (2016, methodology): “For instance, local contrast normalization used to be a standard effective tool in vision problems (Pinto et al., 2008; Jarrett et al., 2009; Sermanet et al., 2012; Le, 2013).”
Recursive deep learning for natural language processing and computer vision (2014, methodology): “LCN was inspired by computational neuroscience and is used to contrast features within a feature map, as well as across feature maps at the same spatial location (Jarrett et al., 2009; Pinto et al., 2008; Le et al., 2012).”
Detecting Avocados to Zucchinis: What Have We Done, and Where Are We Going? (2013, background): “Others have provided insight into dataset design [14, 18, 7].”
Human vs. Computer in Scene and Object Recognition (2014, background): “The computer vision community has made rapid advances in several areas recently. n some restricted cases (e.g., where variability is low), computers even outperform humans for tasks such as frontal-view face recognition, fingerprint recognition, change detection, etc.”
Seeing it all: Convolutional network layers map the function of the human visual system (2017, background): “15 edge detectors, this is almost certainly untrue for the mechanism by which mammalian 295 brains process faces due to the strong invariance and selectivity properties with respect to 296 nontrivial transformations that they can undergo (see [42] for a discussion).”
Performance-optimized hierarchical models predict neural responses in higher visual cortex (2014, methodology): “This parameter space expresses an inclusive version of the hierarhical feedforward network concept and contains models similar to that used in many previous studies for different parameter values (19, 23, 24, 25).”
Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks (2018, methodology): “We have previously shown that, unlike some photographic image sets, synthetic images of the types we used here are critical to separating some types of computer vision systems from humans (Pinto et al., 2008).”
Vehicle Type Classification Using a Semisupervised Convolutional Neural Network (2015, background): “The purpose of the local contrast normalization layer is to enforce local competitions between one neuron and its neighbors, which is motivated by the computational neuroscience [34], [35].”
Real-world visual statistics and infants' first-learned object names (2017, background): “However, this characterization does not fit contemporary understanding of visual object recognition [28, 64].”
Recurrent Processing during Object Recognition (2013, background): “…(Lupyan and Spivey, 2008; Lupyan et al., 2010; Lupyan, 2012), providing a mapping between how a small, uniﬁed set of biological mechanisms can potentially solve one of the most difﬁcult and important computational problems that the brain is known to solve (Marr, 1982; Pinto et al., 2008).”
Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition (2015, background): “Such functionalities of pooling have a biological motivation as they mimic parts of mammals’ vision systems [53], [108].”