PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition
By Brandon RichardWebster, Samuel E. Anthony, Walter J. Scheirer. 2016
77 citations recorded by the source corpus.
By providing substantial amounts of data and standardized evaluation protocols, datasets in computer vision have helped fuel advances across all areas of visual recognition. But even in light of breakthrough results on recent benchmarks, it is still fair to ask if our recognition algorithms are doing as well as we think they are. The vision sciences at large make use of a very different evaluation regime known as Visual Psychophysics to study visual perception. Psychophysics is the quantitative examination of the relationships between controlled stimuli and the behavioral responses they elicit in experimental test subjects. Instead of using summary statistics to gauge performance, psychophysics directs us to construct item-response curves made up of individual stimulus responses to find perceptual thresholds, thus allowing one to identify the exact point at which a subject can no longer reliably recognize the stimulus class. In this article, we introduce a comprehensive evaluation framework for visual recognition models that is underpinned by this methodology. Over millions of procedurally rendered 3D scenes and 2D images, we compare the performance of well-known convolutional neural networks. Our results bring into question recent claims of human-like performance, and provide a path forward for correcting newly surfaced algorithmic deficiencies.
Subfields: Cognitive Neuroscience
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
- Wolfram Burgard
- Ben Upcroft
- Chen Fang
- Dieter Fox
- Ting-Li Chen
- Katherine L. Hermann
- Pieter Abbeel
- Jianchao Yang
Citations in the corpus
- The limits and potentials of deep learning for robotics (2018, background): “…is through a computational pipeline that is able to perturb 2D natural images or 3D rendered scenes at a massive scale (e.g., millions of images per image transformation being studied) and submit them to a model, generating an item-response curve from the resulting recognition scores [93].”
- EnlightenGAN: Deep Light Enhancement Without Paired Supervision (2019, result): “This is based on our empirical observation that the classification results by VGG models are not very sensitive when we manipulate the input pixel intensity range, which is concurred by another recent study [38].”
- The Origins and Prevalence of Texture Bias in Convolutional Neural Networks (2019, background): “Although their performance in several of these tasks approaches that of humans, recent findings show that CNNs differ in intriguing ways from human vision, indicating fundamental deficiencies in our understanding of these models [87, 70, 32, 23, 76, 37, 3, 44, 46, 1, 48].”