TensoRF: Tensorial Radiance Fields

By Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, Hao Su. European Conference on Computer Vision, 2022

1,785 citations recorded by the source corpus.

We present TensoRF, a novel approach to model and reconstruct radiance fields. Unlike NeRF that purely uses MLPs, we model the radiance field of a scene as a 4D tensor, which represents a 3D voxel grid with per-voxel multi-channel features. Our central idea is to factorize the 4D scene tensor into multiple compact low-rank tensor components. We demonstrate that applying traditional CP decomposition -- that factorizes tensors into rank-one components with compact vectors -- in our framework leads to improvements over vanilla NeRF. To further boost performance, we introduce a novel vector-matrix (VM) decomposition that relaxes the low-rank constraints for two modes of a tensor and factorizes tensors into compact vector and matrix factors. Beyond superior rendering quality, our models with CP and VM decompositions lead to a significantly lower memory footprint in comparison to previous and concurrent works that directly optimize per-voxel features. Experimentally, we demonstrate that TensoRF with CP decomposition achieves fast reconstruction (<30 min) with better rendering quality and even a smaller model size (<4 MB) compared to NeRF. Moreover, TensoRF with VM decomposition further boosts rendering quality and outperforms previous state-of-the-art methods, while reducing the reconstruction time (<10 min) and retaining a compact model size (<75 MB).

View Paper: https://www.semanticscholar.org/paper/d267731870c41d977c9d51195c1e2fd018846949

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Structured 3D Latents for Scalable and Versatile 3D Generation (2024, unknown): “These vectors represent the CP-decomposition [3] of a local 8 3 radiance volume V ∈ R 8 × 8 × 8 × 4 : The last dimension of V , which has a size of 4, contains the color and density information.”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Hybrid methods [4, 7, 24, 27, 41, 46] incorporate a neural component within the explicit grid.”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, background): “Motivated by recent advances in explicit-implicit hybrid scene representations [7,8], we generalize BEV by accompanying it with two perpendicular planes to construct three cross-planes perpendicular to each other.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “Along the lines of MLP-less radiance field, TensorRF [55] (March 2022) also used neural-network-free volume rendering.”
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures (2022, background): “This embedding can either be a dense voxel grid [23, 43], a sparse voxel grid [41], a low-rank decomposition of a voxel grid [9], a point-based representation [50], or a multi-resolution hash map [34].”
NeRF: Representing scenes as neural radiance ﬁelds for view synthesis (unknown): “Two recent representative works in this area are Instant Neural Graphics Primitives [ 43 ] which uses a multiresolution hash-grid and TensorRF [ 9 ] which does a tensor deocmposition of the feature grid.”
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (2023, methodology): “Neural Radiance Fields (NeRF [29]) and its variants [2, 6, 30] have shown impressive results in reconstructing static scenes from multi-view observations.”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, methodology): “Visualization of the implicit-based SOTA methods SparseNeRF (S) [42] and FreeNeRF (F) [52] when transferred to fast backbones Instant-NGP (iNGP) [26], TensoRF [6], and 3D Gaussian Splatting (3DGS) [18].”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, unknown): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “The idea of tensor factorization has been studied in multiple works [9, 10, 50, 52] to further reduce data redundancy and speed-up rendering.”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, methodology): “Neural representations, as highlighted in prior works [9,62–64, 69, 84, 86 ], present a promising foundation 3D Generation.”
Text-to-3D using Gaussian Splatting (2023, unknown): “After works have emerged to improve NeRF in reconstruction quality [7, 8, 76], handling large-scale [13, 40, 68, 81] and dynamic scenes [3, 47, 51, 60, 73], improving training [10, 44, 67, 79], rendering [25, 56, 80] speed and facilitating down-stream tasks [17, 46, 78, 83, 84].”
LRM: Large Reconstruction Model for Single Image to 3D (2023, unknown): “Different NeRF variants (Chan et al., 2022; Chen et al., 2022a; M¨uller et al., 2022; Sun et al., 2022) typically differ from each other in terms of the choice of the spatial encoding and the size of the MLP.”
HexPlane: A Fast Representation for Dynamic Scenes (2023, methodology): “Recently, rendering speeds during both stages are substantially reduced by using explicit-implicit representations [5, 7, 15, 36, 47, 71].”
Compact 3D Gaussian Representation for Radiance Field (2023, background): “Subsequent research efforts have been directed toward the reduction of the memory footprint while maintaining or even enhancing the performance quality by grid factorization [6, 7, 11, 13, 15, 31], hash grids [30], grid quantization [38, 40] or pruning [10, 35].”
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (2023, methodology): “Some subsequent works have attempted to improve the speed, such as employing well-tailored data structures (Chen et al., 2022; Sun et al., 2022; Hu et al., 2022; Chen et al., 2023), discarding large MLP (Fridovich-Keil et al., 2022), or adopting hash encodings (M¨uller et al., 2022).”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, unknown): “Another group trades space for speed by incorporating explicit and localized neural representations [8, 15, 17, 27, 36, 60, 67, 75, 82, 83, 96, 101].”