GAN Steerability without optimization

By Nurit Spingarn-Eliezer, Ron Banner, Tomer Michaeli. 2020

60 citations recorded by the source corpus.

Recent research has shown remarkable success in revealing "steering" directions in the latent spaces of pre-trained GANs. These directions correspond to semantically meaningful image transformations e.g., shift, zoom, color manipulations), and have similar interpretable effects across all categories that the GAN can generate. Some methods focus on user-specified transformations, while others discover transformations in an unsupervised manner. However, all existing techniques rely on an optimization procedure to expose those directions, and offer no control over the degree of allowed interaction between different transformations. In this paper, we show that "steering" trajectories can be computed in closed form directly from the generator's weights without any form of training or optimization. This applies to user-prescribed geometric transformations, as well as to unsupervised discovery of more complex effects. Our approach allows determining both linear and nonlinear trajectories, and has many advantages over previous methods. In particular, we can control whether one transformation is allowed to come on the expense of another (e.g. zoom-in with or without allowing translation to keep the object centered). Moreover, we can determine the natural end-point of the trajectory, which corresponds to the largest extent to which a transformation can be applied without incurring degradation. Finally, we show how transferring attributes between images can be achieved without optimization, even across different categories.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, background): “A few methods [118], [119], [129], [130] recent show that interpretable directions for image synthesis can be directly obtained in closed forms without training or optimization.”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “Given the inverted images, we can leverage GAN-based editing methods [Härkönen et al. 2020; Kocasari et al. 2022; Shen and Zhou 2021; Spingarn et al. 2021; Voynov and Babenko 2020] to manipulate the style code w.”
Designing an encoder for StyleGAN image manipulation (2021, background): “Others have proposed self-supervised approaches that find latent space directions corresponding to a specific image transformation, such as zoom or rotation [17, 29, 36].”
Pivotal Tuning for Latent-based Editing of Real Images (2021, methodology): “2020], others [Jahanian et al. 2019; Plumerault et al. 2020; Spingarn-Eliezer et al. 2020] find meaningful directions in a self-supervised fashion, and, finally, recent works present unsupervised methods to achieve the same goal [Härkönen et al.”
Low-Rank Subspaces in GANs (2021, background): “To identify these semantically meaningful subspaces, existing approaches fall into two folds, i.e. , supervised [13, 30, 18, 24] and unsupervised [33, 15, 28, 31].”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, background): “Others have explored self-supervised approaches [32, 58, 66], the mixing of latent codes to produce local edits [12,14,33], and the use of contrastive language-image (CLIP) models [59] to achieve new editing capabilities [21,54,82].”
LARGE: Latent-Based Regression through GAN Semantics (2021, methodology): “Many recent works have proposed methods to interpret the semantics encoded in this latent space and apply them to image editing [2, 20, 37, 45, 50, 57, 62].”