The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

By Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, Oliver Wang. 2018

18,743 citations recorded by the source corpus.

While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Media Technology, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

  1. Richard Zhang
  2. Jun-Yan Zhu
  3. Yang Song
  4. Alexei A. Efros
  5. Antonio Torralba
  6. Bolei Zhou
  7. Eli Shechtman
  8. Daniel Cohen-Or

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Ming-Hsuan Yang
  3. Dacheng Tao
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrea Vedaldi
  8. Andrew Zisserman

In Media Technology

  1. Radu Timofte
  2. Luc Van Gool
  3. Ming-Hsuan Yang
  4. Chao Dong
  5. Lei Zhang
  6. Kai Zhang
  7. Wangmeng Zuo
  8. Chen Change Loy

In Evaluation & Benchmarks

  1. Jitendra Malik
  2. Luc Van Gool
  3. Antonio Torralba
  4. Vladlen Koltun
  5. Alexei A. Efros
  6. Trevor Darrell
  7. Ming-Hsuan Yang
  8. Chen Change Loy

Citations in the corpus