Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?

By Rameen Abdal, Yipeng Qin, Peter Wonka. 2019

1,206 citations recorded by the source corpus.

We propose an efficient algorithm to embed a given image into the latent space of StyleGAN. This embedding enables semantic image editing operations that can be applied to existing photographs. Taking the StyleGAN trained on the FFHD dataset as an example, we show results for image morphing, style transfer, and expression transfer. Studying the results of the embedding algorithm provides valuable insights into the structure of the StyleGAN latent space. We propose a set of experiments to test what class of images can be embedded, how they are embedded, what latent space is suitable for embedding, and if the embedding is semantically meaningful.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Analyzing and Improving the Image Quality of StyleGAN (2019, background): “Previous research [1, 9] suggests that instead of finding a common latent code w, the results improve if a separatew is chosen for each layer of the generator.”
Generative Adversarial Networks (2021, neutral): “Style transfer using GANs has shown promising results in various domains, s.a. fashion design, art, and photography [19].”
StyleGAN-NADA (2021, background): “Thus, by considering editing directions in theW+ space [1] – the latent space where each layer of StyleGAN can be provided with a different code wi ∈ W – we can identify which layers are most strongly tied to a given change.”
GAN Inversion: A Survey (2021, methodology): “Some studies [18], [21] analyze the separability and semantics of both W and Z spaces.”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “Standard approaches for inverting G use either latent optimization [Abdal et al. 2019; Creswell and Bharath 2019; Karras et al. 2020b] or an encoder [Alaluf et al. 2021; Perarnau et al. 2016; Tov et al. 2021].”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “We compare to edits stemming from edit vectors computed by Image2StyleGAN (Abdal et al. 2019) and InterfaceGAN (Hrknen et al. 2020) and refer to these methods as linear models.”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “Most previous works have considered only distortion [1, 27, 31] for evaluating the reconstruction.”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “Generally, inversion methods either directly optimize the latent vector to minimize the error for the given image [25, 8, 1, 2], train an encoder to map the given image to the latent space [31, 8, 32, 12, 29], or use a hybrid approach combining both [4, 42].”
PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models (2020, methodology): “We formalize this through the downscaling loss, to explicitly penalize a proposed SR image for deviating from its LR input (similar loss terms have been proposed in [1],[21]).”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “Unlike other works that analyze the (intermediate) latent space W or W+ [1], we examine StyleSpace, the space spanned by the channel-wise style parameters, denoted S .”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, methodology): “[1] obtained excellent embedding results by optimizing the latent codes in an enhanced W latent space instead of the initial Z latent space.”
Editing in Style: Uncovering the Local Semantics of GANs (2020, methodology): “Extension to semantic editing of real images can be envisioned by combining our approach with the recent work that embeds natural images into the latent space of StyleGAN [1, 17].”
Style Intervention: How to Achieve Spatial Disentanglement with Style-based Generators? (2020, background): “Image2StyleGAN [1] and its improvement [2] proposes to investigate the extended latent space previous to affine layers, which aims at providing more freedom for GAN inversion.”
Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains (2020, methodology): “There are however many methods for encoding an image into the latent space of StyleGAN [1] [12] [5] [10] which could also be used.”
Swapping Autoencoder for Deep Image Manipulation (2020, methodology): “In each block, we show both the reconstruction and swapping for ours, Im2StyleGAN [1, 40], and StyleGAN2 [41], as well as the style transfer results of STROTSS [45] and WCT(2) [78].”