Editing in Style: Uncovering the Local Semantics of GANs

By Edo Collins, Raja Bala, Bob Price, Sabine Süsstrunk. 2020

297 citations recorded by the source corpus.

While the quality of GAN image synthesis has improved tremendously in recent years, our ability to control and condition the output is still limited. Focusing on StyleGAN, we introduce a simple and effective method for making local, semantically-aware edits to a target output image. This is accomplished by borrowing elements from a source image, also a GAN output, via a novel manipulation of style vectors. Our method requires neither supervision from an external model, nor involves complex spatial morphing operations. Instead, it relies on the emergent disentanglement of semantic objects that is learned by StyleGAN during its training. Semantic editing is demonstrated on GANs producing human faces, indoor scenes, cats, and cars. We measure the locality and photorealism of the edits produced by our method, and find that it accomplishes both.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Many works explore how to utilize the latent space of a pretrained generator for image manipulation [6, 45, 50].”
Rewriting a Deep Generative Model (2020, background): “Recently, thanks to rapid advances of deep generative models [26,45,31], learning-based image synthesis and editing methods have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “and as it does not employ an intermediate latent space which is essential for GAN-based image editing [Abdal et al. 2021; Collins et al. 2020; Patashnik et al. 2021; Wu et al. 2021].”
Designing an encoder for StyleGAN image manipulation (2021, background): “[7] perform local semantic editing by borrowing a subset of the latent code from other samples.”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “[7] perform local semantic editing by manipulating corresponding components of the latent code.”
One Shot Face Swapping on Megapixels (2021, background): “As StyleGAN2 has a layer-wise representation [9, 22, 23], it is heuristically feasible to manipulate latent codes by any network that operates on vectors.”
Pretraining is All You Need for Image-to-Image Translation (2022, background): “There are prior attempts [37, 9, 56, 49, 51, 33, 31] to leverage a pretrained model as generative prior for conditional image synthesis, image editing and restoration.”
Labels4Free: Unsupervised Segmentation using StyleGAN (2021, methodology): “Comparison with ReDo [2] and Editing in Style [3] Methods We further compare our method with ReDo [2] and Editing in Style [3].”
Explaining in Style: Training a GAN to explain a classifier in StyleSpace (2021, methodology): “This approach has been used to create powerful interfaces for image editing and data visualization [3, 26, 7, 28, 27, 37].”
Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains (2020, methodology): “This would be a similar idea to the conditional interpolation introduced by Collins et al. [3] whereas here we would be interpolating between models rather than latent vectors.”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, background): “Thanks to their semantically rich latent representations, many works have utilized these models to facilitate diverse and expressive editing through latent space manipulations [4, 6, 9, 12, 24, 38, 44, 48, 56].”
Navigating the GAN Parameter Space for Semantic Image Editing (2020, methodology): “using them as an implicit learnable objective [20, 17, 13, 5, 18, 36, 37], employing them as high-quality image priors [35, 2, 9, 22, 23], manipulating their internal representations for visual editing purposes [3, 6].”
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, background): “Deep learning is now powering the majority of computer vision applications ranging from autonomous driving [93, 73] and medical imaging [78, 38] to image editing [69, 15, 98, 88, 70, 74].”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, background): “Specifically, StyleGAN generators [33–36] synthesize exceed-ingly realistic images, and enable editing [4,5,9,15,42,48, 83], and image-to-image translation [46,47,55], particularly in well-structured domains.”
A Free Viewpoint Portrait Generator with Dynamic Styling (2020, methodology): “use linear SVM [28, 4] to classify the latent code with a corresponding semantic label.”
Ensembling with Deep Generative Views (2021, background): “eral interesting properties to emerge, where the generator learns meaningful variations in data without requiring an explicit training objective to do so [21, 53, 12].”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “Style-based GANs [31, 48, 66, 67] learn the unsupervised separation of high-level attributes and stochastic variation in the generated images.”