PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

By Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, Cynthia Rudin. 2020

648 citations recorded by the source corpus.

The primary aim of single-image super-resolution is to construct a high-resolution (HR) image from a corresponding low-resolution (LR) input. In previous approaches, which have generally been supervised, the training objective typically measures a pixel-wise average distance between the super-resolved (SR) and HR images. Optimizing such metrics often leads to blurring, especially in high variance (detailed) regions. We propose an alternative formulation of the super-resolution problem based on creating realistic SR images that downscale correctly. We present a novel super-resolution algorithm addressing this problem, PULSE (Photo Upsampling via Latent Space Exploration), which generates high-resolution, realistic images at resolutions previously unseen in the literature. It accomplishes this in an entirely self-supervised fashion and is not confined to a specific degradation operator used during training, unlike previous methods (which require training on databases of LR-HR image pairs for supervised learning). Instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is formalized through the “downscaling loss,” which guides exploration through the latent space of a generative model. By leveraging properties of high-dimensional Gaussians, we restrict the search space to guarantee that our outputs are realistic. PULSE thereby generates super-resolved images that both are realistic and downscale correctly. We show extensive experimental results demonstrating the efficacy of our approach in the domain of face super-resolution (also known as face hallucination). Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously possible.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “There are also potential problematic aspects of image generation models, as demonstrated with biases found in the PULSE model (Menon et al., 2020) (see Section 6) and in model correlations with human biases found in social psychology (Steed & Caliskan, 2021), as well as with possible uses of such…”
Image Super-Resolution via Iterative Refinement (2021, methodology): “Metric PULSE [17] FSRGAN [14] Regression SR3”
Alias-Free Generative Adversarial Networks (2021, background): “They have been used for a variety of applications, including image editing [47, 53, 42, 21, 38, 3], domain translation [66, 36, 58, 41], and video generation [55, 15, 23].”
Palette: Image-to-Image Diffusion Models (2021, methodology): “We avoid pixel-level metrics like PSNR and SSIM as they are not reliable measures of sample quality for difficult tasks that require hallucination, like recent super-resolution work, where [Dahl et al. 2017; Ledig et al. 2017; Menon et al. 2020] observe that PSNR and SSIM tend to prefer blurry regre”
Designing an encoder for StyleGAN image manipulation (2021, background): “[25] solve single-image super resolution by taking a low-resolution image and searching the latent space for a high-resolution version of the image using direct optimization.”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “PULSE [33] approaches this task in an unsupervised manner.”
SinSR: Diffusion-Based Image Super-Resolution in a Single Step (2023, methodology): “Besides, GAN-based methods also achieve great success in terms of perceptual quality [9, 13, 17, 27, 34].”
Implicit Diffusion Models for Continuous Super-Resolution (2023, neutral): “Because PULSE generates SR objects incorrectly, its PSNR and LPIPS are significantly lower than other methods.”
Improved Transformer for High-Resolution GANs (2021, background): “For example, there are known issues around bias and fairness, either in the representation of generated images [39] or the implicit encoding of stereotypes [55].”
GAN Prior Embedded Network for Blind Face Restoration in the Wild (2021, methodology): “[34] proposed a self-supervised approach that traverses the HR natural image manifold, searching for images that can downscale to the original LR image.”
AgileGAN (2021, background): “Our motivation is also related to PULSE [Menon et al. 2020], which involves GAN latent space exploration for photo upsampling.”
When, Why, and Which Pretrained GANs Are Useful? (2022, background): “…semantic editing (Isola et al., 2017; Zhu et al., 2018; Shen et al., 2020; Voynov & Babenko, 2020), image processing (Pan et al., 2020; Ledig et al., 2017; Menon et al., 2020), video generation (Wang et al., 2018a), producing high-quality synthetics (Zhang et al., 2021; Voynov et al., 2020).”
GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution (2020, methodology): “PULSE [27] mGANprior [11] DGP [28] Similarity 0.”
Denoising Diffusion Restoration Models (2022, neutral): “Multiple such methods were suggested, mainly focusing on generative adversarial networks (GANs) [7, 11, 33].”
Pretraining is All You Need for Image-to-Image Translation (2022, background): “There are prior attempts [37, 9, 56, 49, 51, 33, 31] to leverage a pretrained model as generative prior for conditional image synthesis, image editing and restoration.”