Using latent space regression to analyze and leverage compositionality in GANs

By Lucy Chai, Jonas Wulff, Phillip Isola. 2021

82 citations recorded by the source corpus.

In recent years, Generative Adversarial Networks have become ubiquitous in both research and public perception, but how GANs convert an unstructured latent code to a high quality output is still an open question. In this work, we investigate regression into the latent space as a probe to understand the compositional properties of GANs. We find that combining the regressor and a pretrained generator provides a strong image prior, allowing us to create composite images from a collage of random image parts at inference time while maintaining global consistency. To compare compositional properties across different generators, we measure the trade-offs between reconstruction of the unrealistic input and image quality of the regenerated samples. We find that the regression approach enables more localized editing of individual image parts compared to direct editing in the latent space, and we conduct experiments to quantify this independence effect. Our method is agnostic to the semantics of edits, and does not require labels or predefined concepts during training. Beyond image composition, our method extends to a number of related applications, such as image inpainting or example-based image editing, which we demonstrate on several GANs and datasets, and because it uses only a single forward pass, it can operate in real-time. Code is available on our project page: https://chail.github.io/latent-composition/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Numerous editing approaches invert [1–3, 12] or encode [8, 50, 63] images into a latent space (e.”
ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement (2021, background): “Diverging from the above approach, recent works [32, 36, 6, 9] have proposed end-to-end methods for leveraging the high-quality images generated by GANs for various image-to-image translation and image editing tasks.”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, methodology): “To preserve identity, some recent methods [13], [49] incorporate a dedicated face recognition loss [50] to measure the cosine similarity between the”
Ensembling with Deep Generative Views (2021, methodology): “As the shifting operation may introduce unknown pixels around the edges of the image, the encoder and optimization step are both performed with a masking input to account for these missing pixels [11].”
Diffusion Self-Guidance for Controllable Image Generation (2023, background): “This degree of control has been explored in the past in smaller scale settings [5, 8, 19, 24, 37, 40], but has not been convincingly demonstrated with modern large-scale diffusion models [26, 32, 38].”
BlobGAN: Spatially Disentangled Scene Representations (2022, background): “The internal compositionality of GANs can be leveraged to harmonize images [15, 13] or perform a limited set of edits on objects in a scene [6, 97, 101].”