Generating Images with Perceptual Similarity Metrics based on Deep Networks

By Alexey Dosovitskiy, Thomas Brox. 2016

1,209 citations recorded by the source corpus.

Image-generating machine learning models are typically trained with loss functions based on distance in the image space. This often leads to over-smoothed results. We propose a class of loss functions, which we call deep perceptual similarity metrics (DeePSiM), that mitigate this problem. Instead of computing distances in the image space, we compute distances between image features extracted by deep neural networks. This metric better reflects perceptually similarity of images and thus leads to better results. We show three applications: autoencoder training, a modification of a variational autoencoder, and inversion of deep convolutional networks. In all cases, the generated images look sharp and resemble natural images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Our perceptual compression model is based on previous work [23] and consists of an autoencoder trained by combination of a perceptual loss [102] and a patch-based [32] adversarial objective [20, 23, 99].”
Making Convolutional Networks Shift-Invariant Again (2019, background): “…hidden units (Girshick et al., 2014; Zhou et al., 2015), actively maximizing hidden units (Mordvintsev et al., 2015), and mapping features back into pixel space (Zeiler & Fergus, 2014; H´enaff & Si-moncelli, 2016; Mahendran & Vedaldi, 2015; Dosovitskiy & Brox, 2016a;b; Nguyen et al., 2017).”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “[27, 10], amongst others, attribute this tendency to the usage of a similarity metric in pixel space.”
Projected GANs Converge Faster (2021, neutral): “Commonly, the losses are combined additively [10, 14, 32, 52, 64].”
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, methodology): “To recover an image from x 1 , we utilize a CNN decoder G , such that the reverse model is speciﬁed as ) Here, f rec denotes the perceptual similarity metric [20, 29, 16, 77] (known as LPIPS) and D φ denotes a patch-based adversarial discriminator [22].”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “Deep network activations are used as the statistics of interest, as they have been shown to correspond well with human perceptual judgments [80] and are often used as training objectives [20, 26, 33].”
Image Super-Resolution via Iterative Refinement (2021, background): “[26] observed that these conventional automated evaluation measures do not correlate well with human perception when”
Taming Transformers for High-Resolution Image Synthesis (2020, methodology): “To do so, we propose VQGAN, a variant of the original VQVAE, and use a discriminator and perceptual loss [36, 26, 35, 16] to keep good perceptual quality at increased compression rate.”
Object-Centric Image Generation from Layouts (2020, neutral): “Perceptual loss Adding a perceptual loss (Dosovitskiy and Brox 2016; Gatys, Ecker, and Bethge 2016; Johnson, Alahi, and Fei-Fei 2016) to our model improved results slightly.”
Diagnosing and Enhancing VAE Models (2019, background): “Experiments from Section 5 reveal that this procedure can generate high-quality crisp samples, avoiding the blurriness often attributed to VAE models in the past (Dosovitskiy and Brox, 2016; Larsen et al., 2015).”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “These methods measure distance in VGG feature space as a “perceptual loss” for image regression problems [23, 14].”
Image-to-Image Translation with Conditional Adversarial Networks (2016, methodology): “A large body of literature has considered losses of this kind, with methods including conditional random fields [9], the SSIM metric [53], feature matching [13], nonparametric losses [34], the convolutional pseudo-prior [54], and losses based on matching covariance statistics [27].”
Generative Visual Manipulation on the Natural Image Manifold (2016, background): “Previous work [5,25] suggests that using deep neural network activations leads to a reconstruction of perceptually meaningful details.”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, methodology): “VGGNet [47]) for perceptual losses [11,22] (Figs.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “s using a pretrained network (VGG-16 relu4 2 [47]), rather than over RGB pixel values. Computing distances in deep feature space, like this, is also sometimes referred to as using a “perceptual loss” [8,23]. BiGAN/ALI [9,7] Unconditional GANs [16] learn a generator G: Z!X, that maps a random noise z”
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (2016, methodology): “y Li and Wand [38] for style transfer and Yeh et al. [64] for inpainting. Bruna et al. [5] minimize the squared error in the feature spaces of VGG19 [49] and scattering networks. Dosovitskiy and Brox [13] use loss functions based on Euclidean distances computed in the feature space of neural network”
Rewriting a Deep Generative Model (2020, neutral): “the perceptual distance between images [38,18,84].”
Adversarially Learned Inference (2016, background): “Recently, efforts have aimed to bridge the gap between VAEs and GANs, to learn generative models with higher-quality samples while learning an efﬁcient inference network (Larsen et al. , 2015; Lamb et al. , 2016; Dosovitskiy and Brox, 2016).”
Toward Controlled Generation of Text (2017, background): “…discriminators that assess generated samples and back-propagate gradients to guide the optimization of generators in a holistic manner, as shown to be highly effective in continuous image generation and representation modeling (Chen et al., 2016; Larsen et al., 2016; Dosovitskiy & Brox, 2016).”