HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing

By Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, Amit H. Bermano. 2021

336 citations recorded by the source corpus.

The inversion of real images into StyleGAN's latent space is a well-studied problem. Nevertheless, applying existing approaches to real-world scenarios remains an open challenge, due to an inherent trade-off between reconstruction and editability: latent space regions which can accurately represent real images typically suffer from degraded semantic control. Recent work proposes to mitigate this trade-off by fine-tuning the generator to add the target image to well-behaved, editable regions of the latent space. While promising, this fine-tuning scheme is impractical for prevalent use as it requires a lengthy training phase for each new image. In this work, we introduce this approach into the realm of encoder-based inversion. We propose HyperStyle, a hypernetwork that learns to modulate StyleGAN's weights to faithfully express a given image in editable regions of the latent space. A naive modulation approach would require training a hypernetwork with over three billion parameters. Through careful network design, we reduce this to be in line with existing encoders. HyperStyle yields reconstructions comparable to those of optimization techniques with the near real-time inference capabilities of encoders. Lastly, we demonstrate HyperStyle's effectiveness on several applications beyond the inversion task, including the editing of out-of-domain images which were never seen during training. Code is available on our project page: https://yuval-alaluf.github.io/hyperstyle/.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Imagic: Text-Based Real Image Editing with Diffusion Models (2022, methodology): “Multiple techniques for applying such manipulations on real images were suggested, including optimization-based methods [1, 2, 25], encoder-based methods [4, 48, 64], and methods adjusting the model per input [5, 9, 15, 49].”
HyperInverter: Improving StyleGAN Inversion via Hypernetwork (2021, background): “[4] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H.”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, background): “Some of the questions we attempt to answer are: How does the disentanglement of the latent representations in StyleGAN3 compare to StyleGAN2? Do the techniques devised for identifying latent editing controls still work? Inversion is a fundamental task required for editing real images, and has been e”
Re-Imagen: Retrieval-Augmented Text-to-Image Generator (2022, methodology): “Previously, GANs (Goodfellow et al., 2014) have been used to achieve signiﬁcant performance on image editing (Zhu et al., 2016; Abdal et al., 2019; Zhu et al., 2020; Roich et al., 2021; Tov et al., 2021; Wang et al., 2022; Alaluf et al., 2022).”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “There it is typical to apply a delicate tuning of a generative model to better reconstruct speciﬁc faces or scenes (Bau et al., 2019; Roich et al., 2021; Alaluf et al., 2021; Dinh et al., 2022; Cao et al., 2022; Nitzan et al., 2022).”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “Tremendous efforts [2, 5, 36, 50, 56, 58] have been made along this way, mostly relying on the semantic latent space of generative models, e.”
High-fidelity GAN Inversion with Padding Space (2022, background): “Some also [3,10,38] try to fine-tune the pre-trained generator, leading to better inversion but much slower speed.”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “This process, known as inversion , has recently drawn considerable attention for GANs, e.g., [51, 1, 3, 35, 50, 43, 45, 47], but has not yet been fully addressed for text-guided diffusion models.”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “Successful results of HyperNetwork are also reported in image generation using generative adversarial networks [1, 10] and other machine learning tasks [51].”