BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

By Thu Nguyen-Phuoc, Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, Niloy J. Mitra. 2020

253 citations recorded by the source corpus.

We present BlockGAN, an image generative model that learns object-aware 3D scene representations directly from unlabelled 2D images. Current work on scene representation learning either ignores scene background or treats the whole scene as one object. Meanwhile, work that considers scene compositionality treats scene objects only as image patches or 2D layers with alpha maps. Inspired by the computer graphics pipeline, we design BlockGAN to learn to first generate 3D features of background and foreground objects, then combine them into 3D features for the wholes cene, and finally render them into realistic images. This allows BlockGAN to reason over occlusion and interaction between objects' appearance, such as shadow and lighting, and provides control over each object's 3D pose and identity, while maintaining image realism. BlockGAN is trained end-to-end, using only unlabelled single images, without the need for 3D geometry, pose labels, object masks, or multiple views of the same scene. Our experiments show that using explicit 3D features to represent objects allows BlockGAN to learn disentangled representations both in terms of objects (foreground and background) and their properties (pose and identity).

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, background): “generation [172], object modeling [173], and human reconstruction [174], [175], [176], [177].”
On the Binding Problem in Artificial Neural Networks (2020, background): “…et al. (2020) demonstrates how a (spatial) mixture model that combines the output of multiple deep neural networks is able to learn to generate images as compositions of individual objects and a background (see also Nguyen-Phuoc et al., 2020; Ehrhardt et al., 2020; Niemeyer and Geiger, 2020).”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Regular structures such as voxelgrids or octrees [13, 48, 14, 88, 94, 89, 43, 81, 107, 59, 60] or unsupervised decomposition [73, 85, 109, 83] enable editability via manipulation of localized parameters.”
BlobGAN: Spatially Disentangled Scene Representations (2022, background): “Similar to us, an active line of work explores adding 3D inductive biases [53, 50, 51], but individual object manipulation has largely focused on simple diagnostic scenes [33].”
SofGAN: A Portrait Image Generator with Dynamic Styling (2020, methodology): “Other controllable neural modeling methods [Chen and Zhang 2019; Mescheder et al. 2019; Nguyen-Phuoc et al. 2019, 2020; Park et al. 2019a] employ traditional implicit geometry representations such as the signed distance field (SDF) and occupancy field (OF) but under neural network approximations.”
ROOTS: Object-Centric Representation and Rendering of 3D Scenes (2020, background): “Another line of work (Nguyen-Phuoc et al., 2020; Liao et al., 2020) learns object-aware 3D scene representations for generative adversarial networks (Goodfellow et al.”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, background): “Some works [7, 16, 38, 45, 63] integrate NeRF with GANs to design 3D-aware generators.”
Generative Adversarial Transformers (2021, background): “Most related to our work are certain GAN models for conditional and unconditional visual synthesis: A few meth-ods [21, 33, 54, 64] utilize multiple replicas of a generator to produce a set of image layers, that are then combined through alpha-composition.”
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis (2021, background): “Voxel-based GANs (Henzler et al., 2019; Nguyen-Phuoc et al., 2019; 2020) lack ﬁne details in the results due to the voxel resolution restriction.”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Coordinate-based networks have also been explored in the context of generative frameworks [7, 9, 20, 38, 39, 50].”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “The evolution of these techniques, from Generative Adversarial Networks (GANs)[3, 9, 10, 24, 25, 27] to diffusion-based frameworks[33], marks a notable progression.”
MVDream: Multi-view Diffusion for 3D Generation (2023, background): “With Generative Adversarial Networks (GANs) yielding improved results in image synthesis, numerous studies have investigated 3D-aware GANs (Nguyen-Phuoc et al., 2019; 2020; Niemeyer & Geiger, 2021; Deng et al., 2022; Chan et al., 2022; Gao et al., 2022).”
Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis (2020, methodology): “With the advance in novel-view synthesis, such as the recent work of BlockGAN (Nguyen-Phuoc et al., 2020) and RGBD-GAN (Noguchi & Harada, 2020), our framework could be potentially extended to deal with broader types of images.”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, methodology): “By introducing differentiable 3D projection and using trainable layers that operate in the 3D and 2D feature space, several recent methods [4, 18, 42, 43, 56, 62] are able to model the geometry and appearance of 3D scenes from 2D images.”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, methodology): “Baselines: We compare against voxel-based Platonic-GAN [32], BlockGAN [64], and HoloGAN [63], and radiance ﬁeld-based GRAF [77] (see Sec. of the methods).”
GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020, background): “HoloGAN [35] and some related works [27,36] learn a low-dimensional 3D feature combined with a learnable 3D-to-2D projection.”
RealFusion 360° Reconstruction of Any Object from a Single Image (2023, methodology): “Several authors have proposed to extract 3D models from 2D image generators, originally using GANs [2, 6, 8, 24, 25, 47].”
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020, methodology): “Visual Object Networks [69] and PrGANs [11] learn to synthesize 2D images by first generating a voxelized 3D shape using a 3D-GAN [64] and then projecting it into 2D. HoloGAN [41] and BlockGAN [42] have extended the system by incorporating a volumetric but implicit 3D representation.”