StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows

By Rameen Abdal, Peihao Zhu, Niloy J. Mitra, Peter Wonka. 2020

602 citations recorded by the source corpus.

High-quality, diverse, and photorealistic images can now be generated by unconditional GANs (e.g., StyleGAN). However, limited options exist to control the generation process using (semantic) attributes while still preserving the quality of the output. Further, due to the entangled nature of the GAN latent space, performing edits along one attribute can easily result in unwanted changes along other attributes. In this article, in the context of conditional exploration of entangled latent spaces, we investigate the two sub-problems of attribute-conditioned sampling and attribute-controlled editing. We present StyleFlow as a simple, effective, and robust solution to both the sub-problems by formulating conditional exploration as an instance of conditional continuous normalizing flows in the GAN latent space conditioned by attribute features. We evaluate our method using the face and the car latent space of StyleGAN, and demonstrate fine-grained disentangled edits along various attributes on both real photographs and StyleGAN generated images. For example, for faces, we vary camera pose, illumination variation, expression, facial hair, gender, and age. Finally, via extensive qualitative and quantitative comparisons, we demonstrate the superiority of StyleFlow over prior and several concurrent works. Project Page and Video: https://rameenabdal.github.io/StyleFlow.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Networks (2021, neutral): “It was proved successful in many domains such as computer vision [14, 29, 25, 38], semantic segmentation [39, 27, 70, 24], time-series synthesis [9, 23], image editing [61, 36, 19, 3, 75], natural language processing [15, 28, 22], text-to-image generation [59, 58, 54], and many more.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “There exists specialized techniques for this task [3,37,44,57], but here we aim to showcase the quality and applicability of our latent space by using the simplest linear operation.”
StyleGAN-NADA (2021, methodology): “Multi-domain editing of a real, inverted image using StyleCLIP [37] (mohawk and surprised), InterFaceGAN [45] (age) and StyleFlow [3] (pose).”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Furthermore, subsequent manipulations are typically carried out by moving along a direction in one of the latent spaces, using a parametric model, such as a 3DMM in StyleRig [45], or a trained normalized ﬂow in StyleFlow [1].”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “Motivated by this, it has become common practice [1, 2, 4, 42, 3] to encode real images into an extended latent space, W+, defined by the concatenation of 18 different 512-dimensional w vectors, one for each input layer of StyleGAN.”
GANSpace: Discovering Interpretable GAN Controls (2020, neutral): “Several methods developed concurrently to our own explore similar or related ideas [23, 15, 24, 20, 1], and comparing or combining approaches may prove useful as well.”
ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement (2021, background): “[3] learn non-linear paths via normalizing flows conditioned on a target attribute.”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, methodology): “(b) gender=female,smile=1,age=55 (Left: StyleFlow, Right: Ours) Figure 2: Conditionally generated images of our method (LACE-ODE) and StyleFlow [1] on the glasses and gender_smile_age of FFHQ, respectively.”
LARGE: Latent-Based Regression through GAN Semantics (2021, methodology): “Many recent works have proposed methods to interpret the semantics encoded in this latent space and apply them to image editing [2, 18, 35, 38, 51, 57, 69].”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, background): “Various works have demonstrated that editing images via non-linear latent paths typically results in more faithful, disentangled edits [4, 27].”
Dual Contrastive Loss and Attention for GANs (2021, background): “processing [46, 71, 44, 45, 77, 62, 102], image manipulation [13, 14, 70, 1, 4, 80], texture synthesis [94, 53, 59], image inpainting [34, 52, 92, 93], and text-to-image generation [68, 99, 100, 74].”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “They either modify the training objective [9, 39, 70] or network architecture [38], or investigate latent spaces of well-engineered and pre-trained generative models [1, 16, 23, 26, 33, 77, 95].”
Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks (2021, methodology): “…that the GAN can be exploited for downstream tasks like unsupervised and few-shot part segmentation (Zhang et al., 2021; Tritrong et al., 2021; Abdal et al., 2021a; Collins et al., 2020; Bielski & Favaro, 2019), extracting 3D models of the objects (Pan et al., 2021; Chan et al., 2020) and…”