Collaborative Learning for Faster StyleGAN Embedding

By Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, Xiaokang Yang. 2020

111 citations recorded by the source corpus.

The latent code of the recent popular model StyleGAN has learned disentangled representations thanks to the multi-layer style-based generator. Embedding a given image back to the latent space of StyleGAN enables wide interesting semantic image editing applications. Although previous works are able to yield impressive inversion results based on an optimization framework, which however suffers from the efficiency issue. In this work, we propose a novel collaborative learning framework that consists of an efficient embedding network and an optimization-based iterator. On one hand, with the progress of training, the embedding network gives a reasonable latent code initialization for the iterator. On the other hand, the updated latent code from the iterator in turn supervises the embedding network. In the end, high-quality latent code can be obtained efficiently with a single forward pass through our embedding network. Extensive experiments demonstrate the effectiveness and efficiency of our work.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, methodology): “[103] propose the embedding network, which consists of two encoders, an identity encoder Eid to extract identity from the input image x, and an attribute Encoder”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “Generally, inversion methods either (i) directly optimize the latent vector to minimize the error for the given image [1, 2, 8, 21, 23], (ii) train an encoder to map the given image to the latent space [13, 28, 31], or (iii) use a hybrid approach combining both [4, 45].”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “Generally, inversion methods either directly optimize the latent vector to minimize the error for the given image [25, 8, 1, 2], train an encoder to map the given image to the latent space [31, 8, 32, 12, 29], or use a hybrid approach combining both [4, 42].”
Paint by Word (2021, background): “Because several powerful editing methods are enabled by GAN latent manipulation, the problem of inverting a GAN is of widespread interest and is the subject of an active line of ongoing research [57, 26, 6, 1, 2, 14, 40, 56, 59, 16].”
ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement (2021, methodology): “Other approaches design an encoder [51, 32, 50, 15, 35, 40] to learn a direct mapping from a given image to its corresponding latent vector.”
Improved StyleGAN Embedding: Where are the Good Latents? (2020, methodology): “Such a two-stage encoding process is also employed by several concurrent works based on encoder networks [6, 29, 10, 5], which to our knowledge first appears in iGAN [30].”
Pivotal Tuning for Latent-based Editing of Real Images (2021, background): “Inversion methods can typically be divided into optimization-based ones — which directly optimize the latent code using a single sample [1, 8, 19, 21], or encoder-based ones — which train an encoder over a large number of samples [13, 23, 28].”
E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion (2021, methodology): “Recently, numerous GAN inversion methods [4], [5], [6], [7], [8] have been proposed and shown strong competence in performing meaningful manipulations of human faces in the latent space.”
GAN Inversion for Out-of-Range Images with Geometric Transformations (2021, methodology): “One direction is to train an encoder using a datadriven approach [32, 9, 22].”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, methodology): “…, has been tackled by numerous recent works, either by using: (i) optimization (Abdal et al., 2019; Karras et al., 2020b); or (ii) an encoder (Guan et al., 2020; Pidhorskyi et al., 2020; Richardson et al., 2021; Tov et al., 2021); or (iii) a hybrid approach using both (Zhu et al., 2016;…”
One Shot Face Swapping on Megapixels (2021, methodology): “Firstly, to overcome the information loss in the encoder, we adopt GAN Inversion methods [42, 10, 16, 1, 2, 10, 31, 11, 59] and propose a Hierarchical Representation Face Encoder (HieRFE) to find the complete face representation in an extended latent spaceW.”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, methodology): “StyleGAN2 [36] features several semantically-rich latent spaces, which have been heavily studied and exploited in the context of image manipulation [4,15,26,60,61,65] and image inversion [1,2,10,25,51,85].”
HyperInverter: Improving StyleGAN Inversion via Hypernetwork (2021, background): “Therefore, some hybrid works [5, 6, 14, 57] have been proposed, which combine both above general approaches to partly balance the trade-off of reconstruction quality and inference time.”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, methodology): “Generally, inversion methods either directly optimize the latent vector to minimize the error for a given image [1, 2, 7, 14, 21, 40, 70, 74, 75], train an encoder over a large number of samples to learn a mapping from an image to its latent representation [5,22,22,30, 36, 43, 49, 50, 53, 61, 65], o”
High-Fidelity GAN Inversion for Image Attribute Editing (2021, background): “[11] designed a collaborative learning scheme for encoder and optimization iterator.”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “PSNR [48, 156] measures the ratio between the maximum possible value referred to as the power of a signal and the power of the distorted noise that affects the quality of its representation.”