Image2StyleGAN++: How to Edit the Embedded Images?

By Rameen Abdal, Yipeng Qin, Peter Wonka. 2019

593 citations recorded by the source corpus.

We propose Image2StyleGAN++, a flexible image editing framework with many applications. Our framework extends the recent Image2StyleGAN in three ways. First, we introduce noise optimization as a complement to the W+ latent space embedding. Our noise optimization can restore high frequency features in images and thus significantly improves the quality of reconstructed images, e.g. a big increase of PSNR from 20 dB to 45 dB. Second, we extend the global W+ latent space embedding to enable local embeddings. Third, we combine embedding with activation tensor manipulation to perform high quality local edits along with global semantic edits on images. Such edits motivate various high quality image editing applications, e.g. image reconstruction, image inpainting, image crossover, local style transfer, image editing using scribbles, and attribute level feature transfer. Examples of the edited images are shown across the paper for visual inspection.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Networks (2021, neutral): “Style transfer using GANs has shown promising results in various domains, s.a. fashion design, art, and photography [19].”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “Studying the structure of the StyleGAN1/2, we choose to apply the given vector w ′ at the di erent indices of theW+(Abdal et al. 2019, 2020; Karras et al. 2019b) (R18×512) space depending on the nature of the edit, e.g., we would expect the lighting change to be in the later layers of the StyleGAN…”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “Generally, inversion methods either (i) directly optimize the latent vector to minimize the error for the given image [1, 2, 8, 21, 23], (ii) train an encoder to map the given image to the latent space [13, 28, 31], or (iii) use a hybrid approach combining both [4, 45].”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “Generally, inversion methods either directly optimize the latent vector to minimize the error for the given image [25, 8, 1, 2], train an encoder to map the given image to the latent space [31, 8, 32, 12, 29], or use a hybrid approach combining both [4, 42].”
Paint by Word (2021, background): “Because several powerful editing methods are enabled by GAN latent manipulation, the problem of inverting a GAN is of widespread interest and is the subject of an active line of ongoing research [53, 26, 6, 1, 2, 14, 36, 52, 55, 16].”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “By utilizing the weights of pretrained generators, several works [1, 2, 9, 11, 22] design different latent optimization methods to do inpainting, style transfer, morphing, colorization, denoising and super resolution.”
VOGUE: Try-On by StyleGAN Interpolation Optimization (2021, methodology): “Related to our method are also conditional GAN networks [23, 4], and GAN editing methods [25, 2, 6, 15, 8, 1].”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, methodology): “Recent works [2, 61, 73, 75] have demonstrated the ex-”
InfinityGAN: Towards Infinite-Resolution Image Synthesis (2021, methodology): “Image outpaint-ing [1, 18, 30, 40] is related to image inpainting [19, 41] and shares similar issues that the generator tends to copy-and-paraphrase the conditional input or create mottled textural samples, leading to repetitive results especially when the outpainted region is large.”
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis (2021, methodology): “Some meth-ods (H¨ark¨onen et al., 2020; Tewari et al., 2020a; Shen et al., 2020; Abdal et al., 2020; Tewari et al., 2020b; Leimk¨uhler & Drettakis, 2021; Shoshan et al., 2021) leverage disentangled properties in the latent space to enable explicit controls, most of which focus on faces.”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, methodology): “In the GAN literature, this inversion is done through either an optimization-based technique (Abdal et al., 2019, 2020; Zhu et al., 2020b; Gu et al., 2020) or by using an encoder (Richardson et al., 2020; Zhu et al., 2020a; Pidhorskyi et al., 2020; Tov et al., 2021).”
Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks (2021, background): “When allowing each element to vary independently, the resulting latent space is called W + space Abdal et al. (2019; 2020); Zhu et al. (2020b).”
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs (2020, methodology): “Following prior work [47], [48], [54], we treat the layer-wise styles (i.”
Zero-shot Image-to-Image Translation (2023, background): “Numerous prior works propose to finetune the GAN model to better match the input image[8, 38, 46], explore different latent spaces [62, 1, 2], invert into multiple layers [19, 40], and utilize latent edit directions [21, 54, 41, 3].”
Improved StyleGAN Embedding: Where are the Good Latents? (2020, methodology): “We also do not use noise embedding as proposed by I2S++ [2] because it is too hard to control with respect to editing quality.”
LOHO: Latent Optimization of Hairstyles via Orthogonalization (2021, methodology): “We begin by observing the objective function proposed in Image2StyleGAN++ (I2S++) [2]: (1 where w is an embedding in the extended latent space W + of StyleGAN, n is a noise vector embedding, M s , M m , and M p are binary masks to specify image regions contributing to the respective losses,…”