Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation

By Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei. 2019

51 citations recorded by the source corpus.

The fundamental challenge of planning for multi-step manipulation is to find effective and plausible action sequences that lead to the task goal. We present Cascaded Variational Inference (CAVIN) Planner, a model-based method that hierarchically generates plans by sampling from latent spaces. To facilitate planning over long time horizons, our method learns latent representations that decouple the prediction of high-level effects from the generation of low-level motions through cascaded variational inference. This enables us to model dynamics at two different levels of temporal resolutions for hierarchical planning. We evaluate our approach in three multi-step robotic manipulation tasks in cluttered tabletop environments given high-dimensional observations. Empirical results demonstrate that the proposed method outperforms state-of-the-art model-based methods by strategically interacting with multiple objects.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Joseph J. Lim
2. Animesh Garg
3. De-An Huang
4. Yuke Zhu
5. Jiankai Sun
6. Bolei Zhou
7. Youngwoon Lee
8. Yifeng Zhu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jitendra Malik
3. Li Fei-Fei
4. Cordelia Schmid
5. Ming-Hsuan Yang
6. Abhinav Kumar Gupta
7. Trevor Darrell
8. Xiaogang Wang

In Control and Systems Engineering

1. Joseph J. Lim
2. Yuke Zhu
3. Animesh Garg
4. Youngwoon Lee
5. Bolei Zhou
6. Yifeng Zhu
7. Kostas Daniilidis
8. Jingyun Yang

Citations in the corpus

• Accelerating Reinforcement Learning with Learned Skill Priors (2020, background): “[5] Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei.”
• Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, methodology): “To mitigate this problem, hierarchical structures have been introduced to neural network methods [8, 22, 24, 25, 27].”
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction (2020, background): “Although real-robot learning has been achieved with fullyobservable MBRL [6]–[11], there has been an increasing demand for robot learning in partially observable environments in which only incomplete information (especially vision) is available.”
• PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks (2021, background): “Another related work is learning dynamics models for model-based RL [9], [10], [11].”
• Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, background): “Fang et al. (2019) shows that a simple jumpy hierarchical prediction method improves planning performance in the real world.”