Goal-Conditioned Imitation Learning using Score-based Diffusion Policies

By Moritz Reuss, Maximilian Xiling Li, Xiaogang Jia, Rudolf Lioutikov. Robotics: Science and Systems Conference, 2023

298 citations recorded by the source corpus.

We propose a new policy representation based on score-based diffusion models (SDMs). We apply our new policy representation in the domain of Goal-Conditioned Imitation Learning (GCIL) to learn general-purpose goal-specified policies from large uncurated datasets without rewards. Our new goal-conditioned policy architecture"$\textbf{BE}$havior generation with $\textbf{S}$c$\textbf{O}$re-based Diffusion Policies"(BESO) leverages a generative, score-based diffusion model as its policy. BESO decouples the learning of the score model from the inference sampling process, and, hence allows for fast sampling strategies to generate goal-specified behavior in just 3 denoising steps, compared to 30+ steps of other diffusion based policies. Furthermore, BESO is highly expressive and can effectively capture multi-modality present in the solution space of the play data. Unlike previous methods such as Latent Plans or C-Bet, BESO does not rely on complex hierarchical policies or additional clustering for effective goal-conditioned behavior learning. Finally, we show how BESO can even be used to learn a goal-independent policy from play-data using classifier-free guidance. To the best of our knowledge this is the first work that a) represents a behavior policy based on such a decoupled SDM b) learns an SDM based policy in the domain of GCIL and c) provides a way to simultaneously learn a goal-dependent and a goal-independent policy from play-data. We evaluate BESO through detailed simulation and show that it consistently outperforms several state-of-the-art goal-conditioned imitation learning methods on challenging benchmarks. We additionally provide extensive ablation studies and experiments to demonstrate the effectiveness of our method for goal-conditioned behavior generation. Demonstrations and Code are available at https://intuitive-robots.github.io/beso-website/

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/1334a47e8f4e4ffd04ff534329d76a5e5cc16f46

Citations in the corpus

• CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, unknown): “Recent studies [15, 50, 53] have introduced diffusion models as an innovative approach for modeling robotic actions.”
• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, unknown): “Though, similar architectures for action modeling with diffusion have been proposed before [7, 36, 40].”
• 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, unknown): “…works use diffusion models for learning such state-conditioned action distributions for robot manipulation policies from demonstrations [54, 11, 60] and show they outperform deterministic or other alternatives, such as variational autoencoders [47], mixture of Gaussians † Equal contribution…”
• Diffusion policy: Visuomotor policy learning via action diffusion (2023, unknown): “Concurrent to us, Pearce et al. (2023), Reuss et al. (2023), and Hansen-Estruch et al. (2023) have conducted a complimentary analysis of diffusion-based policies in simulated environments.”