Perceptual Losses for Real-Time Style Transfer and Super-Resolution
By Justin Johnson, Alexandre Alahi, Li Fei-Fei. 2016
11,604 citations recorded by the source corpus.
We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a per-pixel loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing perceptual loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results.
Subfields: Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Alexei A. Efros
- Stefano Ermon
- Daniel Cohen-Or
- Yoshua Bengio
- Ming-Hsuan Yang
Citations in the corpus
- Do Better ImageNet Models Transfer Better? (2018, methodology): “Network architectures measured against this dataset have fueled much progress in computer vision research across a broad array of problems, including transferring to new datasets [17, 56], object detection [32], image segmentation [27, 7] and perceptual metrics of images [35].”
- Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, methodology): “We have experimented with two different ways of using the perceptual loss, the feature reconstruction loss (the Euclidean distance between feature maps in the classifier) and the style loss (the Euclidean distance between the Gram matrices [12]).”
- Evaluating Lossy Compression Rates of Deep Generative Models (2020, methodology): “7 shows the RD curves of GANs, VAEs, and AAEs on the MNIST, using the MSE on the deep features of a CNN as distortion metric.”
- CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, methodology): “Perceptual loss We consider models that directly optimize a perceptual loss [18], with no adversarial training.”
- Projected GANs Converge Faster (2021, methodology): “An established method is the combination of adversarial and perceptual losses [21].”
- Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “We use a combination of logit-laplace loss, `2 loss, perceptual loss (Johnson et al., 2016; Zhang et al., 2018) based on VGG network (Simonyan & Zisserman, 2014) and GAN loss with architecture of StyleGAN discriminator (Karras et al.”
- Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “In contrast, perceptual loss [20] evaluates a distance be-tween features extracted from the predicted and the target images by a base pre-trained network ϕ ( · ) .”
- ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, methodology): “To recover an image from x 1 , we utilize a CNN decoder G , such that the reverse model is specified as ) Here, f rec denotes the perceptual similarity metric [20, 29, 16, 77] (known as LPIPS) and D φ denotes a patch-based adversarial discriminator [22].”
- On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “Deep network activations are used as the statistics of interest, as they have been shown to correspond well with human perceptual judgments [80] and are often used as training objectives [20, 26, 33].”
- Taming Transformers for High-Resolution Image Synthesis (2020, methodology): “To do so, we propose VQGAN, a variant of the original VQVAE, and use a discriminator and perceptual loss [36, 26, 35, 16] to keep good perceptual quality at increased compression rate.”
- Dual Contradistinctive Generative Autoencoder (2020, positive): “Feature reconstruction loss (also referred to as perceptual loss), similar to that in style transfer [32].”
- Object-Centric Image Generation from Layouts (2020, neutral): “Perceptual loss We found that adding a perceptual loss [7,10,22] to our model improved results slightly.”
- Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, background): “the perceptual loss [8] or style loss [9].”
- EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “We include the two losses proposed in [16, 24] commonly known as perceptual loss Lperc and style loss Lstyle.”
- Free-Form Image Inpainting With Gated Convolution (2018, background): “Instead, we propose a variant of generative adversarial networks, named SN-PatchGAN, motivated by global and local GANs [15], MarkovianGANs [21], perceptual loss [17] and recent work on spectral-normalized GANs [24].”
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “These methods measure distance in VGG feature space as a “perceptual loss” for image regression problems [23, 14].”
- Image-to-Image Translation with Conditional Adversarial Networks (2016, methodology): “Many previous solutions [40, 52, 27, 60, 56] to problems in this area have used an encoder-decoder network [24].”
- Unrolled Generative Adversarial Networks (2016, methodology): “We optimize with 3 random starts using LBFGS, which is the optimizer typically used in similar settings such as style transfer (Johnson et al., 2016; Champandard, 2016).”