Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation

By Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, Ping Luo. 2020

448 citations recorded by the source corpus.

Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig. 1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature images, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deep-generative-prior.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “By utilizing the weights of pretrained generators, several works [1, 2, 9, 11, 22] design different latent optimization methods to do inpainting, style transfer, morphing, colorization, denoising and super resolution.”
Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs (2020, background): “One way is to treat pretrained GANs as strong image priors, which can be achieved by either exploiting GANs as a component of their networks [22, 5, 6, 17, 3] or directly editing the latent code to find feasible solutions [1, 2, 11].”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, methodology): “Some recent methods [69], [70], [71] of image restoration found that inversion-based methods often generate images with low-fidelity.”
Zero-shot Image-to-Image Translation (2023, background): “Numerous prior works propose to finetune the GAN model to better match the input image[Bau et al. 2019; Pan et al. 2021; Roich et al. 2021], explore different latent spaces [Abdal et al.”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, methodology): “To apply the manipulation to an input image I r belonging to the same training category, the key is to ﬁrst optimize all generation conditions to inversely project the image to the generation manifold, similar to the latent image manipulation methods [1, 2, 10, 27].”
Towards Real-World Blind Face Restoration with Generative Facial Prior (2021, methodology): “Generative Priors of pretrained GANs [35, 36, 37, 3] is previously exploited by GAN inversion [1, 77, 56, 20], whose primary aim is to find the closest latent codes given an input image.”
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs (2020, methodology): “Following Pan et al. (2020), we adopt the L1 distance of the discriminator features as the distance metric L , which is shown to work better for GAN samples.”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “Several methods have been proposed in this direction, including fine-tuning network weights for each image [7, 38, 46], choosing better or multiple layers to project and edit [1, 2, 19, 60], designing better encoders [45, 57], modeling image corruption and transformations [5, 24], and discovering me”
Flow-based Kernel Prior with Application to Blind Super-Resolution (2021, background): “However, while various image priors have been proposed to describe natural image statistics [5, 6, 7, 20, 36, 44], little attention has been paid on designing the kernel prior.”
Dual Contrastive Loss and Attention for GANs (2021, background): “processing [46, 71, 44, 45, 77, 62, 102], image manipulation [13, 14, 70, 1, 4, 80], texture synthesis [94, 53, 59], image inpainting [34, 52, 92, 93], and text-to-image generation [68, 99, 100, 74].”
Pivotal Tuning for Latent-based Editing of Real Images (2021, methodology): “Then, we replace our losses with the discriminator and regularization terms proposed by Pan et al. [2020]. We fine-tune the generator in a progressive manner, starting from the lower layers to the higher ones, as proposed by Pan et al. [2020]. Since the role on the different layers in StyleGAN does ”
Multi-Stage Progressive Image Restoration (2021, methodology): “Recently, CNN-based restoration methods have achieved state-of-theart results [57, 70, 86, 93, 97].”
Denoising Diffusion Restoration Models (2022, methodology): “We [40], DGP [38], and SNIPS [25].”
Ensembling with Deep Generative Views (2021, background): “Due to the higher variation in multi-class data, image projection (a critical part of generating deep augmentations) is still not sufficiently fast and reliable across the full dataset [23, 37, 35].”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, background): “[47] invert BigGAN [8] by randomly sampling noise vectors, selecting the one that best matches the real image, and optimizing it simultaneously with the generator weights in a progressive manner.”
From Continuity to Editability: Inverting GANs with Consecutive Images (2021, methodology): “To realize real image editing, GAN inversion methods are proposed to inference a latent code of an input image based on the pre-trained GAN [16, 25, 1, 3, 15].”
GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution (2020, background): “Through finetuning the generator during optimization, the result of DGP [28] demonstrates significant improvements in both quality and fidelity.”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “Without requiring the pairs of clean and degraded images, unsupervised image restoration [21, 35, 77, 92] has achieved impressive results, but most of them [21, 77, 92]”
Navigating the GAN Parameter Space for Semantic Image Editing (2020, methodology): “This paradigm has been successfully used for super-resolution [22], visual editing [35, 2], image restoration tasks [9, 23], e.”