Implicit Behavioral Cloning

By Peter R. Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian S. Wong, Johnny Lee, Igor Mordatch, Jonathan Tompson. 2021

679 citations recorded by the source corpus.

We find that across a wide range of robot policy learning scenarios, treating supervised policy learning with an implicit model generally performs better, on average, than commonly used explicit models. We present extensive experiments on this finding, and we provide both intuitive insight and theoretical arguments distinguishing the properties of implicit models compared to their explicit counterparts, particularly with respect to approximating complex, potentially discontinuous and multi-valued (set-valued) functions. On robotic policy learning tasks we show that implicit behavioral cloning policies with energy-based models (EBM) often outperform common explicit (Mean Square Error, or Mixture Density) behavioral cloning policies, including on tasks with high-dimensional action spaces and visual image inputs. We find these policies provide competitive results or outperform state-of-the-art offline reinforcement learning methods on the challenging human-expert tasks from the D4RL benchmark suite, despite using no reward information. In the real world, robots with implicit policies can learn complex and remarkably subtle behaviors on contact-rich tasks from human demonstrations, including tasks with high combinatorial complexity and tasks requiring 1mm precision.

Subfields: Artificial Intelligence, Computer Science Applications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “The Adroit and Kitchen tasks are comparatively less discriminating, and we found that most RL methods perform similarly to imitation learning in these domains (Florence et al., 2021).”
Interactive Language: Talking to Robots in Real Time (2022, methodology): “Language-Table’s simulated environment resembles our real-world tabletop manipulation scenario, which consists of an xArm6 robot, constrained to move in a 2D plane with a cylindrical end-effector as in [54], in front of a smooth wooden board with a fixed set of 8 plastic blocks, comprising 4 colors ”
Continuous Control with Action Quantization from Demonstrations (2021, background): “Another recent line of works has considered the problem of modeling demonstrations using an energy-based model, which is well adapted for multimodalities (Jarrett et al., 2020; Florence et al., 2021).”
Instruction-driven history-aware policies for robotic manipulations (2022, methodology): “While earlier methods for solving robotics tasks such as visual servoing [21, 22] were designed manually, the need to cope with large variations of objects and environments led to the emergence of learning-based neural approaches [23, 24, 25, 26].”
Behavior Transformers: Cloning k modes with one stone (2022, methodology): “We also experiment with the Mixture density networks (MDN) [8] and uniform quantization, as shown in previous works [25, 39].”
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, neutral): “[10], combination of classification and regression objectives [66], or energy-based objectives [18].”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “This follows Algorithm 1 of (Florence et al., 2022).”
One Step Diffusion via Shortcut Models (2024, neutral): “We compare to iterative denoising meth-ods IBC (Florence et al., 2022) and diffusion policy (Chi et al., 2023), as well as one-step methods LSTM-GMM (Mandlekar et al., 2021) and BET (Shafiullah et al., 2022).”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, neutral): “An example of this approach, demonstrated in recent research [37], reframes BC as an energy-based modeling problem [38].”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, neutral): “In addition, we report the average of best-performing checkpoints for robomimic and Push-T tasks to be consistent with the evaluation methodology of their respective original papers (Mandlekar et al., 2021; Florence et al., 2021).”
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, neutral): “Diffusion models have also been used in behavioral cloning and offline RL. Florence et al. [10] and Pearce et al. [36] use energy-based models and diffusion models, respectively, for behavioral cloning.”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, methodology): “• Block-Push Environment [10]: We used the adapted goal-conditioned variant [6].”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, methodology): “Action-based IRL Similar to action-based RLHF, action-based IRL also models human choice based on Q instead of cumulative reward (Ramachandran and Amir, 2007; Neu and Szepesvári, 2009; Florence et al., 2022).”
NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration (2023, neutral): “…this by exploring different action representations, such as autoregressive predictions of quantized actions [24–27], using latent variable models [17, 21], switching to an implicit policy representation [28], and, most recently, using conditional diffusion models for planning and control [29–34].”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, background): “We also note that the cross-entropy based training method from Section 3.4 is closely related to Energy-Based Models (EBMs) [98, 99].”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, neutral): “En-hancements to BC include incorporating history with various architectures [12, 47, 59, 77], new training objectives [10, 18, 35, 63, 104], regularization [71], motor primitives [7, 44, 55, 62, 64, 97], and data preprocessing [81].”
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, methodology): “In contrast, recent IL methods built upon deep neural networks can learn reactive policies from raw demonstration data [20, 2, 3, 21–24].”
Code as Policies: Language Model Programs for Embodied Control (2022, background): “, policies pretrained with behavior cloning or reinforcement learning [19]–[21].”