Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

By Vincent Sitzmann, Michael Zollhoefer, Gordon Wetzstein. 2019

1,421 citations recorded by the source corpus.

Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D-structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the three-dimensional structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth or shape. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. We demonstrate the potential of SRNs by evaluating them for novel view synthesis, few-shot reconstruction, joint shape and appearance interpolation, and unsupervised discovery of a non-rigid face model.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, background): “…volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, 13, 17, 26, 32], texture synthesis [15, 31], shape inference from images [22, 23], and novel view synthesis [27, 29, 35, 37].”
How to Represent Part-Whole Hierarchies in a Neural Network (2021, methodology): “It was inspired by a biological analogy, a mathematical analogy, and recent work on neural scene representations [Ha, 2016, Sitzmann et al., 2019].”
Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering (2022, methodology): “Some methods represent the geometry as the zero level-set of a neural network, and use differentiable surface rendering to optimize the network weights [34, 46, 64].”
Multi-view 3D Reconstruction with Transformers (2021, methodology): “, coordinatebased implicit representation [14, 18, 29, 24, 3], voxel[17, 26] and mesh-based [23] representations.”
Neural Scene Graphs for Dynamic Scenes (2020, methodology): “We quantitatively validate our method with comparisons against Scene Representation Networks (SRNs) [30], NeRF [21], and a modiﬁed variant of NeRF on novel scene and reconstruction tasks.”
NeRF: Representing scenes as neural radiance ﬁelds for view synthesis (positive): “Again, the idea is not entirely new, and builds on pioneering work on deep signed distance functions [ 47 ] and scene representation networks [ 62 ].”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, methodology): “5 we also combine PE with other pre-NeRF methods, such as SRN [56] or DVR [45].”
DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (2023, neutral): “4) the collected video is under the permission of the agents.”
Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, methodology): “As a result, our method is fast at inference and achieves real-time rendering speeds while achieving state-of-the-art image quality across multiple metrics on the standard single-view reconstruction benchmark ShapeNet-SRN [35].”
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (2021, methodology): “This representation has been applied successfully to shape representation [26, 27, 31, 4, 1, 10, 47, 32], novel view synthesis [38, 23, 15, 28, 21, 33, 34, 40, 37] and multi-view 3D reconstruction [46, 29, 17, 12, 22].”
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction (2020, neutral): “[35] introduced neural scene representation networks (SRNs).”
Convolutional Occupancy Networks (2020, methodology): “Implicit models have been adopted for learning implicit representations from images [23, 24, 29, 41], for encoding texture information [30], for 4D reconstruction [28] as well as for primitive-based reconstruction [10, 11, 15, 32].”
Learning Continuous Image Representation with Local Implicit Image Function (2020, methodology): “Its differentiable property also allows backpropagation through the model for neural rendering [42] .”
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (2021, background): “Several works [28, 36, 40, 52, 61] proposed differentiable rendering approaches to learn neural representations from only multiview image supervision.”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Such representations have been explored for modeling shape parts [16, 17], objects [3, 6, 10, 19, 25, 27, 31, 35, 36, 41, 43, 44, 51, 55, 62], or scenes [14, 21, 30, 38, 47, 53].”
ROOTS: Object-Centric Representation and Rendering of 3D Scenes (2020, methodology): “Recent works (Eslami et al., 2018; Kumar et al., 2018; Tobin et al., 2019; Tung et al., 2019; Sitzmann et al., 2019a,b; Singh et al., 2019; Mildenhall et al., 2020; Dupont et al., 2020) have explored learning 3D scene representations from 2D images without 3D supervision.”
Local Implicit Grid Representations for 3D Scenes (2020, methodology): “[33] used such implicit networks to represent neural features instead of occupancies that can be combined with a differentiable ray marching algorithm to produce neural renderings of objects.”
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections (2020, background): “In contrast, volume rendering approaches [20, 25, 32] allow for accurate and consistent reconstructions even with large camera motions, as does NeRF-W.”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, background): “Deep Voxels [46] and the follow up work of scene representation networks [47] accumulate features into a 3D volume forming an unsupervised representation”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “Recent emergence of implicit representations [43, 23] significantly advances the synthesis quality of novel views, whereas they tend to find a degenerate solution when given only very few input views.”