Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering

By Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, Yebin Liu. Computer Vision and Pattern Recognition, 2022

300 citations recorded by the source corpus.

We present Tensor4D, an efficient yet effective approach to dynamic scene modeling. The key of our solution is an efficient 4D tensor decomposition method so that the dynamic scene can be directly represented as a 4D spatio-temporal tensor. To tackle the accompanying memory issue, we decompose the 4D tensor hierarchically by projecting it first into three time-aware volumes and then nine compact feature planes. In this way, spatial information over time can be simultaneously captured in a compact and memory-efficient manner. When applying Tensor4D for dynamic scene reconstruction and rendering, we further factorize the 4D fields to different scales in the sense that structural motions and dynamic detailed changes can be learned from coarse to fine. The effectiveness of our method is validated on both synthetic and real-world scenes. Extensive experiments show that our method is able to achieve high-quality dynamic reconstruction and rendering from sparse-view camera rigs or even a monocular camera. The code and dataset will be released at https://github.com/DSaurus/Tensor4D.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/60ec9d5580bb3266c2cc8606887ac5663c44b461

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

HexPlane: A Fast Representation for Dynamic Scenes (2023, unknown): “Tensor4D [64] shares a similar idea as ours, which represents dynamic scenes with 9 planes and multiple MLPs. D-TensoRF [24] regards dynamic ﬁelds as 5D tensors and applies CP/MM decomposition on them for compact representation.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, unknown): “Extending static NeRF-related representations to dynamic scenes are also being actively explored [4, 12, 28, 38, 42, 47, 48, 56, 71, 77, 80, 87, 88, 90, 91].”
K-Planes: Explicit Radiance Fields in Space, Time, and Appearance (2023, methodology): “Others use partially explicit decompositions for video [8,10,13,17,18,28,29,34], usually combining some voxel or spatially decomposed feature grid with one or more MLP components for feature decoding and/or representing scene dynamics.”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, background): “[3, 8, 31] represent further advancements in faster dynamic scene learning.”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Drawing inspiration from seminal NeRF acceleration research, numerous studies on dynamic scene modeling have integrated discrete structures, such as voxel-grids [11, 38], or planes [6, 36].”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, background): “Hook Jumpingjacks Trex BouncingBalls the baseline has difficulty in achieving global optima.”