GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

By Michael Niemeyer, Andreas Geiger. 2020

1,079 citations recorded by the source corpus.

Deep generative models allow for photorealistic image synthesis at high resolutions. But for many applications, this is not enough: content creation also needs to be controllable. While several recent works investigate how to disentangle underlying factors of variation in the data, most of them operate in 2D and hence ignore that our world is three-dimensional. Further, only few works consider the compositional nature of scenes. Our key hypothesis is that incorporating a compositional 3D scene representation into the generative model leads to more controllable image synthesis. Representing scenes as compositional generative neural feature fields allows us to disentangle one or multiple objects from the background as well as individual objects’ shapes and appearances while learning from unstructured and unposed image collections without any additional supervision. Combining this scene representation with a neural rendering pipeline yields a fast and realistic image synthesis model. As evidenced by our experiments, our model is able to disentangle individual objects and allows for translating and rotating them in the scene as well as changing the camera pose.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “(2020) demonstrates how a (spatial) mixture model that combines the output of multiple deep neural networks is able to learn to generate images as compositions of individual objects and a background (see also Nguyen-Phuoc et al., 2020; Ehrhardt et al., 2020; Niemeyer and Geiger, 2020).”
Shape As Points: A Differentiable Poisson Solver (2021, methodology): “Neural implicit representations have been successfully used to represent geometries of objects [10, 16, 18, 35, 40, 44, 47, 50, 57, 59, 62, 63] and scenes [8, 26, 34, 46, 52, 57].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “Like previous works [16, 38, 63], instead of directly predicting x’s RGB, we predict a high-dimensional feature vector F (x) ∈ R(256) for the 3D sampling point x.”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “Recently, neural implicit representations demonstrated promising results for object geometry representation [8, 22, 24, 32, 34–36, 40, 55, 60, 63, 64], scene completion [6, 17, 37], novel view synthesis [23, 25, 38, 66] and also generative modelling [7, 30, 31, 43].”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, methodology): “Following [49], volume rendering in our GAN framework produces feature images, rather than RGB images, because feature images contain more information that can be effectively utilized for the image-space refinement described next.”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, background): “This information allows future works to explore scene compositionality [57], [72], [74] in real-world street scenes.”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, background): “The success of NeRF has also inspired many follow-up works that extend the NeRF to dynamic scenes [30, 33, 8, 40], relighting [3, 39], generative models [37, 25, 4, 13], etc.”
Object Scene Representation Transformer (2022, methodology): “The starting point for our investigations is the Scene Representation Transformer (SRT) [29] as a geometry-free novel view synthesis (NVS) backbone that provides instant novel-scene generalization and scalability to complex datasets.”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, neutral): “Earlier works addressing this problem primarily leverage Generative Adversarial Networks (GANs) [3, 4, 12–14, 18, 33, 34, 46, 73].”
NeRF: Representing scenes as neural radiance ﬁelds for view synthesis (positive): “…graphics conference has given at least one of their best paper awards 5 to articles based on NeRF and extensions, for example, Giraﬀe at CVPR 21 [ 46 ], MipNeRF and Coco3D at ICCV 21 [ 2, 55 ], RefNeRF at CVPR 22 [ 73 ], our own work on Neural Implicit Evolution at ECCV 22 [ 39 ], InstantNGP…”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “3D-Aware Generative Image Synthesis Inspired by the success of neural volume rendering [45] and implicit representations [43, 14], recent work started tackling the problem of 3D-aware image synthesis [7, 57, 49, 26, 25, 76, 8, 51, 58, 67].”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “We render feature images from the model using neural volume rendering [15] of features [16], from the neural field parameterized by the set of feature volumes W and the MLP f .”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, background): “NeRF has drawn wide attention across the vision community [68, 34, 40, 36, 65] due to its simplicity and extraordinary performance.”