Transforming and Projecting Images into Class-conditional Generative Networks

By Minyoung Huh, Richard Zhang, Jun-Yan Zhu, Sylvain Paris, Aaron Hertzmann. 2020

116 citations recorded by the source corpus.

We present a method for projecting an input image into the space of a class-conditional generative neural network. We propose a method that optimizes for transformation to counteract the model biases in generative neural networks. Specifically, we demonstrate that one can solve for image translation, scale, and global color transformation, during the projection optimization to address the object-center bias and color bias of a Generative Adversarial Network. This projection process poses a difficult optimization problem, and purely gradient-based optimizations fail to find good solutions. We describe a hybrid optimization strategy that finds good projections by estimating transformations and class parameters. We show the effectiveness of our method on real images and further demonstrate how the corresponding projections lead to better editability of these images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Paint by Word (2021, background): “Because several powerful editing methods are enabled by GAN latent manipulation, the problem of inverting a GAN is of widespread interest and is the subject of an active line of ongoing research [53, 26, 6, 1, 2, 14, 36, 52, 55, 16].”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, background): “Some concurrent GAN inversion methods [53] demonstrate that gradient-free optimizers find better solutions than gradient-based methods.”
GAN-Supervised Dense Visual Alignment (2021, background): “However, accurate GAN inversion for non-face datasets remains somewhat challenging and slow, despite recent progress [3, 32].”
High-fidelity GAN Inversion with Padding Space (2022, background): “Active attempts broadly fall into two categories, which are optimizationbased [7,11,17,29,32,35] and learning-based [2,33,34,36,40, 45, 51–53].”
Diverse Image Generation via Self-Conditioned GANs (2020, methodology): “Previous GAN inversion methods do not account for class conditioning, so we extend the encoder+optimization hybrid method of [44, 3].”
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation (2021, background): “However, image manipulation by GAN inversion still demands further investigation, because many datasets are still hard to invert due to the limited inversion capability of GAN models [24, 28, 45].”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “…2021), designing better encoders (Richardson et al., 2021; Tov et al., 2021), modeling image corruption and transformations (Anirudh et al., 2020; Huh et al., 2020), and discovering meaningful latent directions (Shen et al., 2020; Goetschalckx et al., 2019; Jahanian et al., 2020; Härkönen et…”
Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation (2020, methodology): “latent codes [44], [45], or a hybrid method of them [15], [46].”
Ensembling with Deep Generative Views (2021, methodology): “Following best practices, we center the car using a bounding box prior to GAN projection, as GAN reconstruction tends to be better on centered objects [23].”
Regularizing Generative Adversarial Networks under Limited Data (2021, unknown): “The GAN models are the corner-stone techniques for numerous vision applications, such as data augmentation [12, 13], domain adaptation [19, 20], image extrapolation [10, 66], image-to-image translation [22, 37, 86], and visual content creation [1, 4, 21, 23, 70, 72].”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “also require GAN inversion, which can be hard for certain datasets [24, 32, 8, 64].”
Zero-shot Image-to-Image Translation (2023, background): “Another category of editing approaches is manipulating the latent space of GANs via image inverting the image and discovering the editing direction [Bau et al. 2021; Huh et al. 2020; Richardson et al. 2020; Wulff and Torralba 2020; Zhu et al. 2020, 2016].”
GAN Inversion for Out-of-Range Images with Geometric Transformations (2021, background): “Recently, in order to enhance the inversion quality, several attempts to widen the latent space have been made [8, 20, 13].”
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (2021, unknown): “[16] and Zhu et al .”
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, methodology): “Optmization-based methods iteratively optimize a reconstruction objective [99, 92, 56, 1, 36, 16, 75, 74] or perform Markov chain Monte Carlo [21], while encoder-”