Perceptual Annotation: Measuring Human Vision to Improve Computer Vision

By Walter J. Scheirer, Samuel E. Anthony, Ken Nakayama, David D. Cox. 2014

74 citations recorded by the source corpus.

For many problems in computer vision, human learners are considerably better than machines. Humans possess highly accurate internal recognition and learning mechanisms that are not yet understood, and they frequently have access to more extensive training data through a lifetime of unbiased experience with the visual world. We propose to use visual psychophysics to directly leverage the abilities of human subjects to build better machine learning systems. First, we use an advanced online psychometric testing platform to make new kinds of annotation data available for learning. Second, we develop a technique for harnessing these new kinds of information-“perceptual annotations”-for support vector machines. A key intuition for this approach is that while it may remain infeasible to dramatically increase the amount of data and high-quality labels available for the training of a given system, measuring the exemplar-by-exemplar difficulty and pattern of errors of human annotators can provide important information for regularizing the solution of the system at hand. A case study for the problem face detection demonstrates that this approach yields state-of-the-art results on the challenging FDDB data set.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Global-and-local attention networks for visual recognition (2018, background): “Previous work has shown that it is possible to augment vision systems with human perceptual judgments on difﬁcult problems including face recognition and localization [23–25], bird categorization [26], action recognition [27], and object detection and segmentation [28–31].”
The limits and potentials of deep learning for robotics (2018, background): “Through experimentation, we have found that human performance vastly exceeds model performance even in cases where a problem has been assumed to be solved (e.g., human face detection [100]).”
PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition (2016, methodology): “[25] introduced the notion of “perceptual annotation” for machine learning, whereby psychophysical measurements are used as weights in a loss function to give a training regime some a priori notion of sample difficulty.”
Can we still avoid automatic face detection? (2016, unknown): “’s “Face in the Branches” detection task [18].”
What are the Visual Features Underlying Human Versus Machine Vision? (2017, background): “Researchers have found that in cases like these human perceptual judgments can augment the performance of vision models [35], resulting in significant gains in face recognition and localization [24, 2, 14], action recognition [31], object detection and segmentation [20, 26, 32].”