Mixture of volumetric primitives for efficient neural rendering

By Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, Jason M. Saragih. ACM Transactions on Graphics, 2021

370 citations recorded by the source corpus.

Real-time rendering and animation of humans is a core function in games, movies, and telepresence applications. Existing methods have a number of drawbacks we aim to address with our work. Triangle meshes have difficulty modeling thin structures like hair, volumetric representations like Neural Volumes are too low-resolution given a reasonable memory budget, and high-resolution implicit representations like Neural Radiance Fields are too slow for use in real-time applications. We present Mixture of Volumetric Primitives (MVP), a representation for rendering dynamic 3D content that combines the completeness of volumetric representations with the efficiency of primitive-based rendering, e.g., point-based or mesh-based methods. Our approach achieves this by leveraging spatially shared computation with a convolutional architecture and by minimizing computation in empty regions of space with volumetric primitives that can move to cover only occupied regions. Our parameterization supports the integration of correspondence and tracking constraints, while being robust to areas where classical tracking fails, such as around thin or translucent structures and areas with large topological variability. MVP is a hybrid that generalizes both volumetric and primitive-based representations. Through a series of extensive experiments we demonstrate that it inherits the strengths of each, while avoiding many of their limitations. We also compare our approach to several state-of-the-art methods and demonstrate that MVP produces superior results in terms of quality and runtime performance.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/43b77c7e55a206c7de3449ec7e898bc6f4986734

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (2023, methodology): “Mixture of Volumetric Primitives [26] uses surface-aligned volumes to achieve fast rendering with high visual fidelity.”
Recent advances in 3D Gaussian splatting (2024, neutral): “It is promising to extract geometry, texture, and lighting with more advanced rendering techniques to facilitate independent editing and build the connection between 3DGS and mesh-based representation to enable efficient editing.”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “…a radiance field as a coordinate-based MLP, this area has expanded with various models, including voxel-based [34, 59, 72], factorization-based [7, 9–11], hash grid-based [42], and point-(or primitive-) based [24, 30, 35, 67] representations among others [4–6 , 74 Feed-forward Reconstruction.”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, methodology): “We optimize the learnable parameters and MLPs with respect to the L 1 loss over rendered pixel colors, with SSIM term [47] L SSIM and volume regularization [28] L vol .”
PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, background): “Also concurrently, Lombardi et al. [25] propose to model data using geometric primitives, and [13, 9, 38] also distill NeRFs to enable real-time rendering.”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, background): “[305] showed a dynamic 3D content rendering representation that combines the”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “Various explicit representations have been investigated, including sparse voxels [15, 37, 63, 71], low-rank components [5, 7, 34, 55], point clouds [4, 22, 84, 97, 101] and others [8,39,47,73,95].”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, methodology): “Instead of directly optimizing the transformation parameters for each control point at different time steps, we employ an MLP Ψ to learn a time-varying transformation field and query the transformation of each control point p k at each timestep t as: Note that in practical implementations, R ti is…”
FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021, methodology): “Finally, there are orthogonal neural rendering strategies capable of fast inference, such as Neural Point-Based Graphics [13], Mixture of Volumetric Primitives [21] or Pulsar [14], which use forms of geometric primitives and rasterization.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “[63] enhance Neural Volumes by decoding a mixture of dynamic geometric primitives from latent code and skipping samples in empty space for efficient ray marching.”
Structured 3D Latents for Scalable and Versatile 3D Generation (2024, positive): “Since original density control schemes are not applicable when Gaussians are predicted by neural networks, we employ regularizations for volume [18] and opacity of the Gaussians to prevent their degeneration, specifically to avoid them becoming excessively large or transparent.”
Advances in Neural Rendering (2021, background): “Mixture of Volumetric Primitives [LSS ∗ 21] is a model for rendering dynamic, animatable virtual humans in real time.”