Diffusion-based Generation, Optimization, and Planning in 3D Scenes
By Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, Song-Chun Zhu. Computer Vision and Pattern Recognition, 2023
334 citations recorded by the source corpus.
We introduce the SceneDiffuser, a conditional generative model for 3D scene understanding. SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning. In contrast to prior work, SceneDiffuser is intrinsically scene-aware, physics-based, and goal-oriented. With an iterative sampling strategy, SceneDiffuser jointly formulates the scene-aware generation, physics-based optimization, and goal-oriented planning via a diffusion-based denoising process in a fully differentiable fashion. Such a design alleviates the discrepancies among different modules and the posterior collapse of previous scene-conditioned generative models. We evaluate the SceneDiffuser on various 3D scene understanding tasks, including human pose and motion generation, dexterous grasp generation, path planning for 3D navigation, and motion planning for robot arms. The results show significant improvements compared with previous models, demonstrating the tremendous potential of the SceneDiffuser for the broad community of 3D scene understanding.
Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering
View Paper: https://www.semanticscholar.org/paper/431031af5dff30525b2ec8cdff45d21a9f78d929
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
In Computer Vision and Pattern Recognition
- Yilun Du
- Max Simchowitz
- Russ Tedrake
- Diego Marti Monso
- Vincent Sitzmann
- Boyuan Chen
- Andreas Geiger
- Antonio Torralba
In Control and Systems Engineering
Citations in the corpus
- Diffusion policy: Visuomotor policy learning via action diffusion (2023, neutral): “In particular, Janner et al. (2022a) and Huang et al. (2023) explore how diffusion models may be used in the context of planning and infer a trajectory of actions that may be executed in a given environment.”
- Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, positive): “They offer diffusion guidance [30, 16] to guide sampling to a desirable sequence, invaluable in decision-making (planning) applications [36, 34].”