HexPlane: A Fast Representation for Dynamic Scenes

By Ang Cao, Justin Johnson. Computer Vision and Pattern Recognition, 2023

779 citations recorded by the source corpus.

Modeling and re-rendering dynamic 3D scenes is a challenging task in 3D vision. Prior approaches build on NeRF and rely on implicit representations. This is slow since it requires many MLP evaluations, constraining real-world applications. We show that dynamic 3D scenes can be explicitly represented by six planes of learned features, leading to an elegant solution we call HexPlane. A HexPlane computes features for points in spacetime by fusing vectors extracted from each plane, which is highly efficient. Pairing a HexPlane with a tiny MLP to regress output colors and training via volume rendering gives impressive results for novel view synthesis on dynamic scenes, matching the image quality of prior work but reducing training time by more than 100×. Extensive ablations confirm our HexPlane design and show that it is robust to different feature fusion mechanisms, coordinate systems, and decoding mechanisms. HexPlane is a simple and effective solution for representing 4D volumes, and we hope they can broadly contribute to modeling spacetime for dynamic 3D scenes.11Project page: https://caoang327.github.io/HexPlane.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a6c3dce03b16c366ba28dfa2232fb68d562e5474

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Bo Dai
2. Qi Tian
3. Linning Xu
4. Deva Ramanan
5. Bastian Leibe
6. Matthias Nießner
7. Tao Lu
8. Mulin Yu

In Computer Graphics and Computer-Aided Design

1. Deva Ramanan
2. Jonathon Luiten
3. Ziyi Yang
4. Georgios Kopanas
5. Bo Dai
6. Qi Tian
7. Linning Xu
8. Bastian Leibe

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Qi Tian
3. Xinggang Wang
4. Dacheng Tao
5. Alan L. Yuille
6. Yi Yang
7. Xiaogang Wang
8. Ming-Hsuan Yang

Citations in the corpus

• GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (2023, methodology): “A common paradigm to model a dynamic scene is directly storing it in an MLP with 4D coordinates as the input [9, 44] or a 4D tensor with the time dimension or spatial dimensions compressed [1, 4, 7, 38].”
• A Survey on 3D Gaussian Splatting (2024, critical): “Advances in diffusion models have driven progress in 4D content generation, yet these methods might produce visually plausible results that violate fundamental physical laws.”
• SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, positive): “Furthermore, by learning a control graph from the scene motion, our control points-based motion representation allows for convenient motion editing, a feature not present in existing methods [1, 5, 11, 37, 38].”
• Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “Extending static NeRF-related representations to dynamic scenes are also being actively explored [4, 12, 28, 38, 42, 47, 48, 56, 71, 77, 80, 87, 88, 90, 91].”
• Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (2023, background): “(b) Methods that represent the scene using an Eulerian representation on a 4D space-time grid, often with various grid decompositions for efficiency such as planar decomposition or hash functions [5, 10, 36].”
• Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Drawing inspiration from seminal NeRF acceleration research, numerous studies on dynamic scene modeling have integrated discrete structures, such as voxel-grids [11, 38], or planes [6, 36].”
• Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians (2024, background): “Various spatial structures have been explored in previous NeRF-based representations, including dense voxel grids [20], [22], sparse voxel grids [17], [21], point clouds [40], multiple compact low-rank tensor components [23], [41], [42], and multi-resolution hash tables [24].”
• Recent advances in 3D Gaussian splatting (2024, neutral): “Quantitative reconstruction results and sizes of 3D scenes after compression are shown in Table 2.”
• VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
• DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “We also find that our baseline method, 3DGS [4], fails in modeling the detail areas, producing unpleasant artifacts, blurring, and unnatural colors.”
• 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, methodology): “Dynamic NeRFs mainly follow two paths: canonical-mapping volume rendering [6, 17, 22, 23, 26] and directly modeling [3, 9, 10].”