The Perception-Distortion Tradeoff

By Yochai Blau, Tomer Michaeli. 2017

1,151 citations recorded by the source corpus.

Image restoration algorithms are typically evaluated by some distortion measure (e.g. PSNR, SSIM, IFC, VIF) or by human opinion scores that quantify perceived perceptual quality. In this paper, we prove mathematically that distortion and perceptual quality are at odds with each other. Specifically, we study the optimal probability for correctly discriminating the outputs of an image restoration algorithm from real images. We show that as the mean distortion decreases, this probability must increase (indicating worse perceptual quality). As opposed to the common belief, this result holds true for any distortion measure, and is not only a problem of the PSNR or SSIM criteria. However, as we show experimentally, for some measures it is less severe (e.g. distance between VGG features). We also show that generative-adversarial-nets (GANs) provide a principled way to approach the perception-distortion bound. This constitutes theoretical support to their observed success in low-level vision tasks. Based on our analysis, we propose a new methodology for evaluating image restoration methods, and use it to perform an extensive comparison between recent super-resolution algorithms.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “Our proposed metric is also related to the classifier-based tests (Blau & Michaeli, 2018; Lopez-Paz & Oquab, 2016).”
SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “We train SoundStream with a mix of losses to achieve both signal reconstruction ﬁdelity and perceptual quality, following the principles of the perception-distortion trade-off discussed in [46].”
Lifelong Generative Modeling (2017, background): “While the Frechet distance, negative ELBO and IS log-likelihood estimate provide a glimpse into model performance, there exists no conclusive metric that captures the quality of unsupervised generative models [124, 108] and active research suggests a direct trade-off between perceptual quality and m”
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (2018, background): “This challenge is the first SR competition that evaluates the performance in a perceptual-quality aware manner based on [22], where the authors claim that distortion and perceptual quality are at odds with each other.”
Designing an encoder for StyleGAN image manipulation (2021, background): “To define a proper reconstruction, we distinguish between two properties: (i) distortion and (ii) perceptual quality [5].”
SinGAN: Learning a Generative Model From a Single Natural Image (2019, methodology): “Following [5], we report distortion (RMSE) and perceptual quality (NIQE [40], lower is better) on BSD100 [35].”
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data (2021, methodology): “We then use the trained PSNR-oriented model as an initialization of the generator, and train the Real-ESRGAN with a combination of L1 loss, perceptual loss [19] and GAN loss [13, 25, 4].”
PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration (2020, methodology): “In some recent works, NR-IQA and FR-IQA methods are combined to measure IR algorithms [4].”
The Unreasonable Effectiveness of Texture Transfer for Single Image Super-resolution (2018, background): “This was illustrated in [10, 11] and recently mathematically analyzed in [12].”
Exploring CLIP for Assessing the Look and Feel of Images (2022, methodology): “However, it is well-known (Blau and Michaeli 2018; Zhang et al. 2018a) that such metrics have a weak correlation to human perception.”
MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment (2022, methodology): “, deblur, denoise, super-resolution) and some works focus on measuring image restoration algorithms using IQA methods [2, 60].”
Diffusion Probabilistic Modeling for Video Generation (2022, background): “Blau and Michaeli [62] theoretically showed that generative models are typically in conﬂict between data reconstruction tasks and achieving a high degree of realism (deﬁned as matching the target distribution unconditionally, without artifacts).”
Low-Light Image and Video Enhancement Using Deep Learning: A Survey (2021, methodology): “, NIQE [88], perceptual index (PI) [88], [89], [90], LOE [37], and SPAQ [91] to quantitatively compare different methods.”
Source Coding of Audio Signals with a Generative Model (2020, background): “As also discussed in [23,24], there is a non-trivial tradeoff between preservation of a waveform match and preservation of the distribution of the signal.”
Introducing the Perception-Distortion Tradeoff into the Rate-Distortion Theory of General Information Sources (2018, background): “Such a subjective evaluation can mathematically be modeled as a hypothesis testing [1].”
The 2018 PIRM Challenge on Perceptual Image Super-Resolution (2018, methodology): “Evaluation The evaluation scheme is based on [1], which proposed to evaluate image restoration algorithms on the perception-distortion plane (see Fig.”
Diffusion Posterior Sampling for General Noisy Inverse Problems (2022, background): “This can be explained by the perception-distortion tradeoff phenomena (Blau & Michaeli, 2018), where preserving high frequency details may actually penalize the reconstructions from having better distortion metrics.”
Learning to Maintain Natural Image Statistics (2018, methodology): “As suggested in [30] we compute both structural similarity (SSIM [37]) to the ground-truth and perceptual quality (NRQM [45]).”