The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
By Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, Oliver Wang. 2018
18,743 citations recorded by the source corpus.
While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.
Subfields: Biophysics, Computer Vision and Pattern Recognition, Media Technology, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Biophysics
- Richard Zhang
- Jun-Yan Zhu
- Yang Song
- Alexei A. Efros
- Antonio Torralba
- Bolei Zhou
- Eli Shechtman
- Daniel Cohen-Or
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Media Technology
- Radu Timofte
- Luc Van Gool
- Ming-Hsuan Yang
- Chao Dong
- Lei Zhang
- Kai Zhang
- Wangmeng Zuo
- Chen Change Loy
In Evaluation & Benchmarks
- Jitendra Malik
- Luc Van Gool
- Antonio Torralba
- Vladlen Koltun
- Alexei A. Efros
- Trevor Darrell
- Ming-Hsuan Yang
- Chen Change Loy
Citations in the corpus
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Our perceptual compression model is based on previous work [23] and consists of an autoencoder trained by combination of a perceptual loss [102] and a patch-based [32] adversarial objective [20, 23, 99].”
- Scaling Autoregressive Video Models (2019, methodology): “However, these metrics were developed for images and have serious flaws when applied to videos (Wang et al., 2004a; Wang & Li, 2007; Zhang et al., 2018; Lee et al., 2018).”
- The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, background): “We do not need to train many different image-to-image models to get diverse distortions (Zhang et al., 2018; Lee et al., 2020).”
- Natural Adversarial Examples (2019, background): “n to this hard ImageNet test set. 1Introduction Research on the ImageNet benchmark has led to numerous advances in classification [30], object detection [27], segmentation [19], and perceptual metrics [47]. ImageNet classification improvements have are broadly applicable and are highly predictive of i”
- Making Convolutional Networks Shift-Invariant Again (2019, methodology): “We use cosine distance, as common for deep features (Kiros et al., 2015; Zhang et al., 2018).”
- Analyzing and Improving the Image Quality of StyleGAN (2019, methodology): “We measure how well the projection succeeds by computing the LPIPS [50] distance between original and re-synthesized image as D LPIPS [ x , g (˜ g − 1 ( x ))] , where x is the image being analyzed and ˜ g − 1 denotes the approximate projection operation.”
- A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “As a basis for our metric, we use a perceptually-based pairwise image distance [59] that is calculated as a weighted difference between two VGG16 [54] embeddings, where the weights are fit so that the metric agrees with human perceptual similarity judgments.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Recently, VQGAN [13] added adversarial and perceptual losses [68] on top of the VQ-VAE reconstruction task, producing reconstructed images with higher quality.”
- Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “We use a combination of logit-laplace loss, `2 loss, perceptual loss (Johnson et al., 2016; Zhang et al., 2018) based on VGG network (Simonyan & Zisserman, 2014) and GAN loss with architecture of StyleGAN discriminator (Karras et al.”
- Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “We follow the established practice in recent image2image literature and use Learned Perceptual Image Patch Similarity (LPIPS) [63] and Fr´echet inception distance (FID) [15] metrics.”
- High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “VQGAN extends previous work on two-stage generative modeling via discrete representation learning [31, 24] by using a more agressive downsampling scheme; enabled by adding an adversarial realism prior and a perceptual metric [35] to the autoencoder objective in [31].”
- On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “Deep network activations are used as the statistics of interest, as they have been shown to correspond well with human perceptual judgments [80] and are often used as training objectives [20, 26, 33].”
- Taming Transformers for High-Resolution Image Synthesis (2020, background): “λ = ∇GL [Lrec] ∇GL [LGAN] + δ (7) where Lrec is the perceptual reconstruction loss [71], ∇GL [·] denotes the gradient of its input w.”
- Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “We adopt the LPIPS metric [48] in computing the DS,”
- Improved Precision and Recall Metric for Assessing Generative Models (2019, background): “[31], on the other hand, use the intermediate activations of multiple convolutional layers of VGG-16 to define a perceptual metric for image corruptions that they show to correlate well with human judgments.”
- EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, background): “cent works [55, 54, 9] have shown that metrics based on deep features are closer to those based on human perception.”
- Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “NVAE [40] achieves the lowest LPIPS and MSE scores, though it requires orders of magnitude larger latent dimension compared to others.”
- StyleGAN-NADA (2021, methodology): “[35] by clustering the data and computing an average LPIPS [63] distance within the clusters.”
- GAN Inversion: A Survey (2021, methodology): “Typically, (cid:96) ( ) can be based on (cid:96) 1 , (cid:96) 2 , perceptual [26] or LPIPS [27] metrics.”