BlobGAN: Spatially Disentangled Scene Representations

By D. Epstein, Taesung Park, Richard Zhang, Eli Shechtman, Alexei A. Efros. 2022

47 citations recorded by the source corpus.

. We propose an unsupervised, mid-level representation for a generative model of scenes. The representation is mid-level in that it is neither per-pixel nor per-image; rather, scenes are modeled as a collection of spatial, depth-ordered “blobs” of features. Blobs are differentiably placed onto a feature grid that is decoded into an image by a generative adversarial network. Due to the spatial uniformity of blobs and the locality inherent to convolution, our network learns to associate different blobs with different entities in a scene and to arrange these blobs to capture scene layout. We demonstrate this emergent behavior by showing that, despite training without any supervision, our method enables applications such as easy manipulation of objects within a scene ( e.g. moving, removing, and restyling furniture), creation of feasible scenes given constraints ( e.g. plausible rooms with drawers at a particular location), and parsing of real-world images into constituent parts. On a challenging multi-category dataset of indoor scenes, BlobGAN outperforms StyleGAN2 in image quality as measured by FID. See our project page for video results and interactive demo: http://www.dave.ml/blobgan. objects that comprise them. Here, we demonstrate through various image manipulation applications that this ability emerges in our model. Our unsupervised representation allows effortless rearrangement, removal, cloning, and restyling of objects in scenes. We also measure correlation between blob presence and semantic categories as predicted by an off-the-shelf network and thus empirically verify the associations discovered by our model.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Numerous editing approaches invert [1–3, 12] or Recent works have used pretrained text-to-image diffusion models for image editing [5,17,27,38,48].”
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold (2023, methodology): “Recently, the controllability of coarse object position is achieved by introducing intermediate “blobs" [Epstein et al. 2022] or heatmaps [Wang et al. 2022b].”
Diffusion Self-Guidance for Controllable Image Generation (2023, background): “This degree of control has been explored in the past in smaller scale settings [5, 8, 19, 24, 37, 40], but has not been convincingly demonstrated with modern large-scale diffusion models [26, 32, 38].”