Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation

By Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, Daniel Cohen-Or. 2020

1,287 citations recorded by the source corpus.

We present a generic image-to-image translation framework, pixel2style2pixel (pSp). Our pSp framework is based on a novel encoder network that directly generates a series of style vectors which are fed into a pretrained StyleGAN generator, forming the extended $\mathcal{W} + $ latent space. We first show that our encoder can directly embed real images into $\mathcal{W} + $, with no additional optimization. Next, we propose utilizing our encoder to directly solve image-to-image translation tasks, defining them as encoding problems from some input domain into the latent domain. By deviating from the standard "invert first, edit later" methodology used with previous StyleGAN encoders, our approach can handle a variety of tasks even when the input image is not represented in the StyleGAN domain. We show that solving translation tasks through StyleGAN significantly simplifies the training process, as no adversary is required, has better support for solving tasks without pixel-to-pixel correspondence, and inherently supports multi-modal synthesis via the resampling of styles. Finally, we demonstrate the potential of our framework on a variety of facial image-to-image translation tasks, even when compared to state-of-the-art solutions designed specifically for a single task, and further show that it can be extended beyond the human facial domain. Code is available at https://github.com/eladrich/pixel2style2pixel.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Networks (2021, neutral): “One may also use StyleGAN for image editing by calculating the latent vector of a given input image in the styleGAN (this operation is known as styleGAN inversion) and then manipulating this vector for editing the image [75, 60, 1, 68, 2, 65, 64, 54].”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Encoding an image to the GAN’s latent space requires an optimization-based inversion process [28, 58] or an external image encoder [39], which has limited reconstruction fidelity (or yields high-dimensional codes outside the learned manifold).”
StyleGAN-NADA (2021, methodology): “Such a latent space is conductive for tasks such as image editing and image-to-image translation [3, 13, 30, 33, 36].”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Similarity to the input image is controlled by the L2 distance in latent space, and by the identity loss [38]:”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Numerous editing approaches invert [1–3, 12] or encode [8, 51, 64] images into a latent space (e.”
Alias-Free Generative Adversarial Networks (2021, background): “They have been used for a variety of applications, including image editing [47, 53, 42, 21, 38, 3], domain translation [66, 36, 58, 41], and video generation [55, 15, 23].”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “The first technique is to build an encoder network [Richardson et al. 2020] that maps an image into the latent space.”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “Most previous works have considered only distortion [1, 27, 31] for evaluating the reconstruction.”
Paint by Word (2021, background): “Because several powerful editing methods are enabled by GAN latent manipulation, the problem of inverting a GAN is of widespread interest and is the subject of an active line of ongoing research [57, 26, 6, 1, 2, 14, 40, 56, 59, 16].”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, positive): “[26] do image translation by training encoders from sketches or semantic maps into StyleGAN’s W space.”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “, 2021), designing better encoders (Richardson et al., 2021; Tov et al., 2021), modeling image corruption and transformations (Anirudh et al.”
TediGAN: Text-Guided Diverse Face Image Generation and Manipulation (2021, methodology): “To preserve identity, some recent methods [33, 29] incorporate a dedicated face recognition loss [7] to measure the cosine similarity between the output image and its source.”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, methodology): “Prior to DM gaining popularity, zero-shot image editing had been dominated by GAN inversion methods [45], [118], [119], [120], [121], [122], [123] combined with CLIP.”
AgileGAN (2021, methodology): “We compared our hierarchical variational encoder to alternative embedding methods, including PSP encoder [Richardson et al. 2020], in-domain encoder [Zhu et al. 2020] and StyleGAN2 optimization [Karras et al. 2020b], by evaluating with these substitutes.”
GAN Inversion for Out-of-Range Images with Geometric Transformations (2021, methodology): “We compare our method with state-of-the-art methods: Im2StyleGAN [1], StyleGAN2 inversion [16], Pnorm [33], and PSP [22].”
Ensembling with Deep Generative Views (2021, methodology): “To speed up the optimization process, several methods [69, 68, 6, 46] propose to train an encoder to initialize latent optimization.”
Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains (2020, methodology): “There are however many methods for encoding an image into the latent space of StyleGAN [1] [12] [5] [10] which could also be used.”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, methodology): “[55], the authors overcome this challenge by using a heuristic that roughly crops the face before passing the image through the facial recognition network.”