Fast Dynamic Radiance Fields with Time-Aware Neural Voxels

By Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, Qi Tian. ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, 2022

463 citations recorded by the source corpus.

Neural radiance fields (NeRF) have shown great success in modeling 3D scenes and synthesizing novel-view images. However, most previous NeRF methods take much time to optimize one single scene. Explicit data structures, e.g. voxel features, show great potential to accelerate the training process. However, voxel features face two big challenges to be applied to dynamic scenes, i.e. modeling temporal information and capturing different scales of point motions. We propose a radiance field framework by representing scenes with time-aware voxel features, named as TiNeuVox. A tiny coordinate deformation network is introduced to model coarse motion trajectories and temporal information is further enhanced in the radiance network. A multi-distance interpolation method is proposed and applied on voxel features to model both small and large motions. Our framework significantly accelerates the optimization of dynamic radiance fields while maintaining high rendering quality. Empirical evaluation is performed on both synthetic and real scenes. Our TiNeuVox completes training with only 8 minutes and 8-MB storage cost while showing similar or even better rendering performance than previous dynamic NeRF methods. Code is available at https://github.com/hustvl/TiNeuVox.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8f01dff0485488c02b567dac44cdfe9f708cfe70

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, unknown): “Another line of research tackles dynamic view synthesis from monocular videos [22, 23, 29, 30, 52, 55, 61, 69, 70, 74, 85, 86, 94].”
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (2023, unknown): “Several intricate details, including the black bars (Pumarola et al., 2021) 29.51 0.95 0.08 D-NeRF (Pumarola et al., 2021) 29.67 0.95 0.07 TiNeuVox (Fang et al., 2022) 32.67 0.97 0.04 HexPlanes (Cao & Johnson, 2023) 31.04 0.97 0.04 K-Planes-explicit (Fridovich-Keil et al., 2023) 31.05 0.97 -…”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Drawing inspiration from seminal NeRF acceleration research, numerous studies on dynamic scene modeling have integrated discrete structures, such as voxel-grids [11, 38], or planes [6, 36].”
Compact 3D Gaussian Representation for Radiance Field (2023, methodology): “These approaches using explicit voxel grid structures [4, 9, 10, 25, 26, 39] have demonstrated a significant improvement in training speed compared to traditional MLP-based NeRF methods.”
Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (2023, result): “On this benchmark we compare to Particle-NeRF [1], InstantNGP [26], and TiNeuVox [9].”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, methodology): “2.”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, methodology): “In most cases [6, 10, 23, 23, 26], positional encoding [35] is used to expand information.”
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, methodology): “We compare our method with the state-of-the-art neural-based method for dynamic scenes, D-NeRF [46] and TiNeuVox [11] quantitatively and qualitatively.”