ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement

By Yuval Alaluf, Or Patashnik, Daniel Cohen-Or. 2021

382 citations recorded by the source corpus.

Recently, the power of unconditional image synthesis has significantly advanced through the use of Generative Adversarial Networks (GANs). The task of inverting an image into its corresponding latent code of the trained GAN is of utmost importance as it allows for the manipulation of real images, leveraging the rich semantics learned by the network. Recognizing the limitations of current inversion approaches, in this work we present a novel inversion scheme that extends current encoder-based inversion methods by introducing an iterative refinement mechanism. Instead of directly predicting the latent code of a given real image using a single pass, the encoder is tasked with predicting a residual with respect to the current estimate of the inverted latent code in a self-correcting manner. Our residual-based encoder, named ReStyle, attains improved accuracy compared to current state-of-the-art encoder-based methods with a negligible increase in inference time. We analyze the behavior of ReStyle to gain valuable insights into its iterative nature. We then evaluate the performance of our residual encoder and analyze its robustness compared to optimization-based inversion and state-of-the-art encoders. Code is available via our project page: https: //yuval-alaluf.github.io/restyle-encoder/

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleGAN-NADA (2021, methodology): “Specifically, we use a ReStyle encoder [4], pre-trained on the human face domain.”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, neutral): “Some methods learn to perform image manipulation in an end-to-end fashion, by training a network that encodes a given image into a latent representation of the manipulated image [40, 46, 2, 3].”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “Standard approaches for inverting G use either latent optimization [Abdal et al. 2019; Creswell and Bharath 2019; Karras et al. 2020b] or an encoder [Alaluf et al. 2021; Perarnau et al. 2016; Tov et al. 2021].”
E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion (2021, methodology): “pSp [10] e4e [61] ReStyle [45] DNI [8] I2S [4] E2Style-1Stage E2Style-2Stage Input Image”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, methodology): “[6] to gradually reﬁne the predicted inversion via small number of forward passes (e.g., 3 ) through the encoder.”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, methodology): “Multiple techniques for applying such manipulations on real images were suggested, including optimization-based meth-ods [1,2,25], encoder-based methods [4,48,64], and meth-ods adjusting the model per input [5, 9, 15,49].”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2022, methodology): “[1], our hypernetworks use T = 5 iterative steps per batch during training.”
High-Fidelity GAN Inversion for Image Attribute Editing (2021, methodology): “.0636 ± .0010 .872 ± .005 .134 ± .006 156s PTI [27] .0622 ± .0004 .877 ± .003 .132 ± .003 283s pSp [26] .0789 ± .0006 .793 ± .006 .169 ± .002 0.11s Restyle pSp [4] .0729 ± .0005 .823 ± .004 .145 ± .002 0.46s e4e [34] .0919 ± .0008 .742 ± .007 .221 ± .003 0.11s Restyle e 4 e [4] editing, we adopt Int”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, methodology): “We compare our results to the state-of-the-art encoders pSp [53], e4e [61], and ReStyle [5] applied over both pSp and e4e.”
High-fidelity GAN Inversion with Padding Space (2022, methodology): “With the startof-the-art synthesis performance, they are widely used for GAN inversion studies [1,2,21,35,36,40,41,45,51,54].”
Identity-Guided Face Generation with Multi-Modal Contour Conditions (2021, methodology): “The ‘baseline’ term in Table 2 is our baseline that concatenates all three images as inputs and produces the W+ latent code with a single ReStyle encoder.”
LARGE: Latent-Based Regression through GAN Semantics (2021, methodology): “For this end, we show that our method is compatible with off-the-shelf GAN Inversion encoders, such as pSp [46], e4e [62] and Re-Style [3].”
Bridging CLIP and StyleGAN through Latent Alignment for Image Editing (2022, positive): “Besides, ReStyle [4] iteratively inverse an image into w + space by addition on the predicted w + residual and achieved SOTA performance.”
GAN-Supervised Dense Visual Alignment (2021, background): “However, accurate GAN inversion for non-face datasets remains somewhat challenging and slow, despite recent progress [3, 32].”
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation (2021, methodology): “direction (GD), Restyle encoder [3] with pSp [32] is used for StyleGAN-NADA, and IDInvert [50] is used for TediGAN, as in their original papers.”