Eigen-Distortions of Hierarchical Representations

By Alexander Berardino, Valero Laparra, Johannes Ballé, Eero P. Simoncelli. 2017

85 citations recorded by the source corpus.

We develop a method for comparing hierarchical image representations in terms of their ability to explain perceptual sensitivity in humans. Specifically, we utilize Fisher information to establish a model-derived prediction of sensitivity to local perturbations of an image. For a given image, we compute the eigenvectors of the Fisher information matrix with largest and smallest eigenvalues, corresponding to the model-predicted most- and least-noticeable image distortions, respectively. For human subjects, we then measure the amount of each distortion that can be reliably detected when added to the image. We use this method to test the ability of a variety of representations to mimic human perceptual sensitivity. We find that the early layers of VGG16, a deep neural network optimized for object recognition, provide a better match to human perception than later layers, and a better match than a 4-stage convolutional neural network (CNN) trained on a database of human ratings of distorted image quality. On the other hand, we find that simple models of early visual processing, incorporating one or more stages of local gain control, trained on the same database of distortion ratings, provide substantially better predictions of human sensitivity than either the CNN, or any combination of layers of VGG16.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, background): “Kim and Lee [23] Recently, Berardino et al. [6] train networks on FR-IQA, and importantly, assess the ability of deep networks to make predictions on a separate task – predicting most and least perceptually-noticeable directions of distortion.”
Generalisation in humans and deep neural networks (2018, background): “These normalisation processes seem to be crucial for many aspects of robust animal and human vision [46], are predictive for human vision data [21, 64] and have proven useful in the context of computer vision [65, 66].”
Image Quality Assessment: Unifying Structure and Texture Similarity (2020, background): “ maps the reference image xand the distorted image yto “perceptual” representations x~ and ~, respectively. The primary motivation is that perceptual distances are nonuniform in the pixel space [35], [36], and this is the main SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGEN”
The Origins and Prevalence of Texture Bias in Convolutional Neural Networks (2019, background): “Euclidean distance in the representation space of CNN hidden layers correlates well with human perceptual similarity [50, 105], although, when used to generate perceptual distortions, simple biologically inspired models are a better match to human perception [9].”
I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively (2020, methodology): “Berardino et al. (2017) developed a computational method for comparing hierarchical image representations in terms of their ability to explain perceptual sensitivity in humans.”
Angular Visual Hardness (2019, background): “Image degradation: Because CNNs and humans achieve similar accuracy on a wide range of tasks on benchmark datasets, a number of works have investigated similarities and differences between CNNs and human vision [3, 5, 9, 11, 12, 29, 46, 51, 67].”
E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles (2019, methodology): “…features in a convolutional neural network (CNN) trained for image classiﬁcation [24, 16, 12] yields a perceptually meaningful space in which to measure image distance [3], with applications in, e.g., image generation and restoration [13, 6, 14] and performance metrics for generative models [11].”
Targeted perturbations reveal brain-like local coding axes in robustified, but not standard, ANN-based brain models (2025, unknown): “…in humans (Gaziv et al., 2023), others have introduced statistical eigen-distortion tests to compare pairs of models (Feather et al., 2024; Berardino et al., 2017), and some studies leverage ro-bustified models trained with neural data to design perceptible stimuli to drive brain…”