Learning to generate images with perceptual similarity metrics

By Jake Snell, Karl Ridgeway, Renjie Liao, Brett D. Roads, Michael C. Mozer, Richard S. Zemel. 2015

206 citations recorded by the source corpus.

Deep networks are increasingly being applied to problems involving image synthesis, e.g., generating images from textual descriptions and reconstructing an input image from a compact representation. Supervised training of image-synthesis networks typically uses a pixel-wise loss (PL) to indicate the mismatch between a generated image and its corresponding target image. We propose instead to use a loss function that is better calibrated to human perceptual judgments of image quality: the multiscale structural-similarity score (MS-SSIM) [1]. Because MS-SSIM is differentiable, it is easily incorporated into gradient-descent learning. We compare the consequences of using MS-SSIM versus PL loss on training autoencoders. Human observers reliably prefer images synthesized by MS-SSIM-optimized models over those synthesized by PL-optimized models, for two distinct PL measures (L1 and L2 distances). We also explore the effect of training objective on image encoding and analyze conditions under which perceptually-optimized representations yield better performance on image classification. Finally, we demonstrate the superiority of perceptually-optimized networks for super-resolution imaging. We argue that significant advances can be made in modeling images through the use of training objectives that are well aligned to characteristics of human perception.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Media Technology

In Generative Models

Citations in the corpus

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “Great research interest has been drawn on the evaluation of GANs (DeVries et al., 2019; Gurumurthy et al., 2017; Sajjadi et al., 2018; Snell et al., 2017; Xiang & Li, 2017).”
Generating Images with Perceptual Similarity Metrics based on Deep Networks (2016, background): “We are not aware of any work making use of similarity metrics for machine learning, except a recent pre-print of Ridgeway et al. (2015).”
Autoencoding beyond pixels using a learned similarity metric (2015, background): “Measuring the quality of generative models is challenging as current evaluation methods are problematic for larger natural images (Theis et al., 2015).”
High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis (2016, neutral): “Recent studies have used various loss functions for image restoration tasks, for instance, `2 loss, SSIM loss [42, 10, 33], `1 loss [42], perceptual loss [22], and adversarial loss [32].”
Learning Temporal Transformations from Time-Lapse Videos (2016, methodology): “training approaches for this network. p mse: As a baseline, we use pixel-wised mean square error(p MSE) between prediction output and ground truth as the loss function. Previous image generation works[1,2,30,3,27] postulate that pixel-wised l 2 criterion is one cause of output generation blurriness.”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “We tried various loss functions based on different feature extractors, such as SSIM loss [40] and feature reconstruction loss [22].”
Deep Feature Consistent Variational Autoencoder (2016, background): “Considering the shortcoming of pixel-by-pixel loss, [26] replaces pixel-by-pixel loss with multi-scale structural-similarity score (MS-SSIM) and demonstrates that it can better measure human perceptual judgments of image quality.”
Modeling and Transforming Speech Using Variational Autoencoders (2016, background): “In the image domain, where a pixel-wise loss is generally not the best choice, there has been some work in this area [17, 18].”
Improving Unsupervised Defect Segmentation by Applying Structural Similarity to Autoencoders (2018, methodology): “Ridgeway et al. (2015) show that SSIM and the closely related multi-scale version MS-SSIM (Wang et al., 2003) can be used as differentiable loss functions to generate more realistic images in deep architectures for tasks such as superresolution, but do not examine its usefulness for defect…”
On the Evaluation of Conditional GANs (2019, background): “Most of these metrics either focus on the separability between generated images and real images [21, 30, 50, 16], compute the distance between distributions [10, 12, 2], assess sample quality and diversity from conditional or marginal distributions [33, 11, 55], measure the similarity between genera”