Neural Scene Graphs for Dynamic Scenes

By Julian Ost, Fahim Mannan, Nils Thürey, Julian Knodt, Felix Heide. 2020

392 citations recorded by the source corpus.

Recent implicit neural rendering methods have demonstrated that it is possible to learn accurate view synthesis for complex scenes by predicting their volumetric density and color supervised solely by a set of RGB images. However, existing methods are restricted to learning efficient representations of static scenes that encode all scene objects into a single neural network, and they lack the ability to represent dynamic scenes and decompose scenes into individual objects. In this work, we present the first neural rendering method that represents multi-object dynamic scenes as scene graphs. We propose a learned scene graph representation, which encodes object transformations and radiance, allowing us to efficiently render novel arrangements and views of the scene. To this end, we learn implicitly encoded scenes, combined with a jointly learned latent representation to describe similar objects with a single implicit function. We assess the proposed method on synthetic and real automotive data, validating that our approach learns dynamic scenes – only by observing a video of this scene – and allows for rendering novel photo-realistic views of novel scene compositions with unseen sets of objects at unseen poses.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “To overcome this issue, other approaches create scene graphs from videos [13, 28, 33, 36, 45, 49].”
NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields (2022, methodology): “Beyond metric-semantic SLAM, our approach can be used as the mapping engine for high-level scene understanding, such as for building 3D Dynamic Scene Graphs [2, 21, 22].”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “Large scene reconstruction is essential for many applications, including autonomous driving [22, 33, 54], aerial sur-veying [6, 13], and virtual reality, which require photo-realistic visual quality and real-time rendering.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, positive): “We demonstrate the qualitative comparison results with our main competitors NSG [11], EmerNeRF [21] and 3DGS [4] on driving scenes reconstruction of nuScenes. method 3DGS [4], our method achieves higher rendering quality with comparable rendering speed.”
Advances in Neural Rendering (2021, methodology): “Neural Scene Graphs (NSG) [OMT ∗ 21] is a recent method for novel view synthesis from monocular videos recorded while driving (ego-vehicle views).”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “Alternatively, one can specify bounding boxes [30, 49], to allow easy compositing of different objects [51] as well as spatial manipulations and geometry deformations [50].”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, background): “Some other works specify bounding boxes [32, 56], to allow flexible compositing of various objects [59] or combining with physical simulation [21].”
Baking Neural Radiance Fields for Real-Time View Synthesis (2021, background): “NeRF has been remarkably successful for view synthesis, and follow-on works have extended it for generative modeling [7, 40], dynamic scenes [24, 35], non-rigidly deforming objects [14, 36], and relighting [3, 44].”
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields (2021, background): “NeRF has inspired many subsequent works that extend its continuous neural volumetric representation for generative modeling [8, 36], dynamic scenes [23, 33], nonrigidly deforming objects [13, 34], phototourism settings with changing illumination and occluders [26, 43], and reflectance modeling for r”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, methodology): “Reconstructing large-scale environments enables several important use-cases in domains such as autonomous driving [32,44,68] and aerial surveying [14,35].”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Other studies also explored reconstruction with more structured hybrid representations via pipelines specialized to a domain (e.g., trafﬁc scene) [58, 22, 36] or situation (e.g., each object data is independently accessible) [25, 24, 92].”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, neutral): “tionality [57], [72], [74] in real-world street scenes.”
Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering (2021, methodology): “[19] propose to use a neural scene graph to decompose dynamic objects in a street view dataset.”
Urban Radiance Fields (2021, methodology): “Neural Scene Graphs [48] consider novel view synthesis from images provided with the KITTI Dataset [21], and NeRF in the Wild (NeRF-W) [39] does the same for internet photo collections.”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “Representing dynamic scenes by neural radiance ﬁelds is an essential extension of NeRF, enabling numerous real-world applications [30,51,56,70,83,89,96].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “Comparison of image reconstruction quality in dynamic KITTI scenes following the the experiement setup of NSG [40].”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, methodology): “Composing neural feature fields: Inspired by works that composite solid objects [23, 56] into a scene, we first transform object-centric neural fields of the foreground actors to world coordinates with the desired poses (e.”