Behavioral Cloning from Observation

By Faraz Torabi, Garrett Warnell, Peter Stone. 2018

950 citations recorded by the source corpus.

Humans often learn how to perform tasks via imitation: they observe others perform a task, and then very quickly infer the appropriate actions to take based on their observations. While extending this paradigm to autonomous agents is a well-studied problem in general, there are two particular aspects that have largely been overlooked: (1) that the learning is done from observation only (i.e., without explicit action information), and (2) that the learning is typically done very quickly. In this work, we propose a two-phase, autonomous imitation learning technique called behavioral cloning from observation (BCO), that aims to provide improved performance with respect to both of these aspects. First, we allow the agent to acquire experience in a self-supervised fashion. This experience is used to develop a model which is then utilized to learn a particular task by observing an expert perform that task without the knowledge of the specific actions taken. We experimentally compare BCO to imitation learning methods, including the state-of-the-art, generative adversarial imitation learning (GAIL) technique, and we show comparable task performance in several different simulation domains while exhibiting increased learning speed after expert trajectories become available.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “In addition to the work described above, the approach taken in this case study takes inspiration from behavioral cloning [12]–[14], variants of which have been used elsewhere to leverage Transformers for imitation learning [15]–[18].”
Pre-Trained Language Models for Interactive Decision-Making (2022, background): “behavior cloning (Pomerleau, 1991; 1989; Torabi et al., 2018), to train the policy.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “…adopted as a standard approach in robotics [Pomerleau, 1989, Schaal, 1997, Argall et al., 2009, Osa et al., 2017, Ye and Alterovitz, 2017, Aleotti and Caselli, 2006, Lawitzky et al., 2012, Torabi et al., 2018, Le et al., 2017, 2018] and many other areas such as playing games [Mnih et al., 2013].”
The MAGICAL Benchmark for Robust Imitation (2020, methodology): “IL algorithms are commonly tested on Reinforcement Learning (RL) benchmark tasks, such as those from OpenAI Gym [37, 23, 27, 8].”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “As an alternative, after inferring the actions, a mapping from states to the actions can be learned and used to improve the learned model and consequently the policy [Torabi et al. , 2018a].”
Learning Latent Plans from Play (2019, background): “Goal conditioned policies have been explored extensively in the literature for reinforcement learning [23, 44, 36, 52, 3, 32, 48, 8, 58], as well as for control via inverse models [1, 35, 9, 61].”
Playing hard exploration games by watching YouTube (2018, background): “Other approaches include interacting with the environment before introducing the expert demonstrations [38, 31] and goal conditioned policies for high ﬁdelity imitation [29], although these papers typically do not assume domain gap or operate in sparse reward settings.”
Imitation Learning as f-Divergence Minimization (2019, background): “[47] Faraz Torabi, Garrett Warnell, and Peter Stone.”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, methodology): “Henderson et al. (2018) and Torabi et al. (2018b) extend Generative Adversarial Imitation Learning (Ho & Ermon, 2016) to remove the need for action labels.”
Primal Wasserstein Imitation Learning (2020, methodology): “The setup is referred to as Learning from Observation (LfO) (Torabi et al., 2018; Edwards et al., 2019).”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “There is a long history of fruitful research in imitation learning and learning via observation that demonstrates the beneﬁts of exploiting such data [39, 14, 165, 7, 145, 38, 185, 132, 119, 1].”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “cloning [55,8,62,11,76] and inverse reinforcement learning [64,49,1,31,23,5,77,45], but also augmenting the RL training with demonstrations [54,21,78,60,58].”
Learning Navigation Subroutines from Egocentric Videos (2019, neutral): “Contemporary works [23, 24, 25] study the problem of learning from state only trajectories, similar to our work here.”
Provable Representation Learning for Imitation Learning via Bi-level Optimization (2020, methodology): “Though some modelbased methods have been proposed (Torabi et al., 2018; Edwards et al., 2018), these methods lack theoretical guarantees.”
S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning (2021, methodology): “Despite the promise of the framework, current offline RL algorithms continue to under-achieve naive baselines such as Behavorial Cloning (Torabi et al., 2018), as reported by Fu et al.”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, result): “Most similar to our own work, Torabi et al. 25 simultaneously train (1) an inverse dynamics model (IDM), 52 which aims to uncover the underlying action between timesteps given observations of past and future timesteps, e.g. p IDM ( a t | o t , o t +1 ) , and (2) a behavioral cloning (BC) model on…”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “There are promising techniques that can potentially overcome this limitation, such as the Learning-from-Observation (LfO) family of algorithms [123, 122, 115, 31].”
One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL (2018, background): “Torabi et al. (2018) also train an inverse dynamics model to learn an unconditional policy.”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “It spans from low data regime (imitation from a single subsampled trajectory [10]) to high data regime (ORL from large datasets [5]), from having access to states, actions and rewards [11] to having only access to observations [43].”
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (2021, background): “While several prior works have studied imitation without example actions (Liu et al., 2018; Torabi et al., 2018a; Aytar et al., 2018; Torabi et al., 2018b; Edwards et al., 2018; Lee et al.)”