Image Processing Using Multi-Code GAN Prior

By Jinjin Gu, Yujun Shen, Bolei Zhou. 2019

345 citations recorded by the source corpus.

Despite the success of Generative Adversarial Networks (GANs) in image synthesis, applying trained GAN models to real image processing remains challenging. Previous methods typically invert a target image back to the latent space either by back-propagation or by learning an additional encoder. However, the reconstructions from both of the methods are far from ideal. In this work, we propose a novel approach, called mGANprior, to incorporate the well-trained GANs as effective prior to a variety of image processing tasks. In particular, we employ multiple latent codes to generate multiple feature maps at some intermediate layer of the generator, then compose them with adaptive channel importance to recover the input image. Such an over-parameterization of the latent space significantly improves the image reconstruction quality, outperforming existing competitors. The resulting high-fidelity image reconstruction enables the trained GAN models as prior to many real-world applications, such as image colorization, super-resolution, image inpainting, and semantic manipulation. We further analyze the properties of the layer-wise representation learned by GAN models and shed light on what knowledge each layer is capable of representing.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, background): “The upper illustration (a) is from mGANPrior [51] and the lower (b) is from DGP [28].”
Alias-Free Generative Adversarial Networks (2021, background): “They have been used for a variety of applications, including image editing [47, 53, 42, 21, 38, 3], domain translation [66, 36, 58, 41], and video generation [55, 15, 23].”
Interpreting the Latent Space of GANs for Semantic Face Editing (2019, methodology): “To make GANs applicable for real image processing, existing methods proposed to reverse the mapping from the latent space to the image space [27, 39, 24, 5, 16] or learn an additional encoder associated with the GAN training [13, 12, 38].”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “By utilizing the weights of pretrained generators, several works [1, 2, 9, 11, 22] design different latent optimization methods to do inpainting, style transfer, morphing, colorization, denoising and super resolution.”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “Several methods have been proposed in this direction, including fine-tuning network weights for each image [7, 38, 46], choosing better or multiple layers to project and edit [1, 2, 19, 60], designing better encoders [45, 57], modeling image corruption and transformations [5, 24], and discovering me”
GAN Prior Embedded Network for Blind Face Restoration in the Wild (2021, methodology): “(f) HiFaceGAN [47] (g) mGANprior [12] (h) PULSE [34] (i) pSp [38] (j) GPEN (k) Ground truth”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, methodology): “To apply the manipulation to an input image I r belonging to the same training category, the key is to ﬁrst optimize all generation conditions to inversely project the image to the generation manifold, similar to the latent image manipulation methods [1, 2, 10, 27].”
Towards Real-World Blind Face Restoration with Generative Facial Prior (2021, methodology): “mGANprior [20] attempts to optimize multiple codes to improve the reconstruction quality.”
Navigating the GAN Parameter Space for Semantic Image Editing (2020, methodology): “This paradigm has been successfully used for super-resolution [22], visual editing [35, 2], image restoration tasks [9, 23], e.”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “The second category leverages GAN inversions (Zhu et al., 2016; Brock et al., 2017; Abdal et al., 2019; Gu et al., 2020; Wu et al., 2021; Abdal et al., 2020), where a pre-trained GAN is used to invert an input image to a latent representation, which is subsequently modified to generate the edited…”
A Simple Baseline for StyleGAN Inversion (2021, neutral): “Existing GAN inversion methods can be subdivided into optimization-based [1, 2, 15, 34], forwardbased [35, 33, 6, 16] and hybrid [43, 8, 7] approaches.”
Unsupervised 3D Shape Completion through GAN Inversion (2021, background): “This appealing nature of GAN prior has been extensively exploited on various image restoration and manipulation tasks [3, 2, 25, 13].”
Denoising Diffusion Restoration Models (2022, background): “Under this unsupervised setting, priors based on deep neural networks have demonstrated impressive empirical results in various image restoration tasks [40, 50, 43, 38, 15].”
Cycle-Consistent Inverse GAN for Text-to-Image Synthesis (2021, methodology): “[9] employ multiple latent codes to generate feature maps at some intermediate layer of the generator, then compose them with adaptive channel importance to recover the input image.”
Transforming and Projecting Images into Class-conditional Generative Networks (2020, methodology): “hoto Editing [9] proposes to edit a face photo using VAE-GANs [38]. The same image prior from deep generative models has also been used in face editing, image inpainting, colorization, and deblurring [49,61,2,25,55]. Recently, GANPaint [5] proposes to change the semantics of an input image by rst pr”
Towards Robust Blind Face Restoration with Codebook Lookup Transformer (2022, neutral): “It is utilized via different strategies of iterative latent optimization for effective GAN inversion [12, 27] or direct latent encoding of degraded faces [29].”
GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution (2020, methodology): “mGANprior [11] optimizes multiple latent codes to increase the expressiveness of the model.”