Behavior Transformers: Cloning k modes with one stone

By Nur Muhammad (Mahi) Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, Lerrel Pinto. 2022

421 citations recorded by the source corpus.

While behavior learning has made impressive progress in recent times, it lags behind computer vision and natural language processing due to its inability to leverage large, human-generated datasets. Human behaviors have wide variance, multiple modes, and human demonstrations typically do not come with reward labels. These properties limit the applicability of current methods in Offline RL and Behavioral Cloning to learn from large, pre-collected datasets. In this work, we present Behavior Transformer (BeT), a new technique to model unlabeled demonstration data with multiple modes. BeT retrofits standard transformer architectures with action discretization coupled with a multi-task action correction inspired by offset prediction in object detection. This allows us to leverage the multi-modal modeling ability of modern transformers to predict multi-modal continuous actions. We experimentally evaluate BeT on a variety of robotic manipulation and self-driving behavior datasets. We show that BeT significantly improves over prior state-of-the-art work on solving demonstrated tasks while capturing the major modes present in the pre-collected datasets. Finally, through an extensive ablation study, we analyze the importance of every crucial component in BeT. Videos of behavior generated by BeT are available at https://notmahi.github.io/bet

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Model Architectures

Citations in the corpus

Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, neutral): “En-hancements to BC include incorporating history with various architectures [12, 47, 59, 77], new training objectives [10, 18, 35, 63, 104], regularization [71], motor primitives [7, 44, 55, 62, 64, 97], and data preprocessing [81].”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, background): “Prior work has shown that highly expressive policies can better model multi-modal action distributions, leading to improved performance on a variety of tasks [50, 18, 47].”
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, neutral): “1: 3D Diffuser Actor sets a new state-of-the-art on RLBench [32] on a multi-view setup and on CALVIN [49] benchmarks on a zero-shot long-horizon setup.”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, methodology): “CLIPort (Shafiullah et al., 2022) and PerAct (Shridhar et al., 2023) learn language-conditioned manipulation by leveraging CLIP (Rad-ford et al., 2021) to connect visual signals and language instructions.”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “A further two baselines can be considered strong, more complex methods, namely K-means+Residual: as with K-means, but additionally learns a continuous residual on top of each bin prediction, trained via MSE, which was the core innovation of Behaviour Transformers (BeT) (Shafiullah et al., 2022); and”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, neutral): “However, suppose each action in the sequence is predicted as independent multimodal distributions (as done in BC-RNN and BET).”
One Step Diffusion via Shortcut Models (2024, neutral): “We compare to iterative denoising meth-ods IBC (Florence et al., 2022) and diffusion policy (Chi et al., 2023), as well as one-step methods LSTM-GMM (Mandlekar et al., 2021) and BET (Shafiullah et al., 2022).”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, background): “Behavior Transformer (Shafiullah et al., 2022) and Gato (Reed et al.”
Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning (2022, background): “However, as offline datasets are often collected by a mixture of policies, the true behavior policy may exhibit strong multi-modalities, skewness, or dependencies between different action dimensions, which cannot be well modeled by diagonal Gaussian policies (Shafiullah et al., 2022).”