An Algorithmic Perspective on Imitation Learning

By Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Bagnell, Pieter Abbeel, Jan Peters. 2018

1,031 citations recorded by the source corpus.

As robots and other intelligent agents move from simple environments and problems to more complex, unstructured settings, manually programming their behavior has become increasingly challenging and expensive. Often, it is easier for a teacher to demonstrate a desired behavior rather than attempt to manually engineer it. This process of learning from demonstrations, and the study of algorithms to do so, is called imitation learning. This work provides an introduction to imitation learning. It covers the underlying assumptions, approaches, and how they relate; the rich set of algorithms developed to tackle the problem; and advice on effective tools and implementation. We intend this paper to serve two audiences. First, we want to familiarize machine learning experts with the challenges of imitation learning, particularly those arising in robotics, and the interesting theoretical and practical distinctions between it and more familiar frameworks like statistical supervised learning theory and reinforcement learning. Second, we want to give roboticists and experts in applied artificial intelligence a broader appreciation for the frameworks and tools available for imitation learning. We pay particular attention to the intimate connection between imitation learning approaches and those of structured prediction Daume III et al. [2009]. To structure this discussion, we categorize imitation learning techniques based on the following key criteria which drive algorithmic decisions: 1) The structure of the policy space. Is the learned policy a time-index trajectory (trajectory learning), a mapping from observations to actions (so called behavioral cloning [Bain and Sammut, 1996]), or the result of a complex optimization or planning problem at each execution as is common in inverse optimal control methods [Kalman, 1964, Moylan and Anderson, 1973]. 2) The information available during training and testing. In particular, is the learning algorithm privy to the full state that the teacher possess? Is the learner able to interact with the teacher and gather corrections or more data? Does the learner have a (typically a priori) model of the system with which it interacts? Does the learner have access to the reward (cost) function that the teacher is attempting to optimize? 3) The notion of success. Different algorithmic approaches provide varying guarantees on the resulting learned behavior. These guarantees range from weaker (e.g., measuring disagreement with the agent’s decision) to stronger (e.g., providing guarantees on the performance of the learner with respect to a true cost function, either known or unknown). We organize our work by paying particular attention to distinction (1): dividing imitation learning into directly replicating desired behavior (sometimes called behavioral cloning) and learning the hidden objectives of the desired behavior from demonstrations (called inverse optimal control or inverse reinforcement learning [Russell, 1998]). In the latter case, behavior arises as the result of an optimization problem solved for each new instance that the learner faces. In addition to method analysis, we discuss the design decisions a practitioner must make when selecting an imitation learning approach. Moreover, application examples—such as robots that play table tennis [Kober and Peters, 2009], programs that play the game of Go [Silver et al., 2016], and systems that understand natural language [Wen et al., 2015]— illustrate the properties and motivations behind different forms of imitation learning. We conclude by presenting a set of open questions and point towards possible future research directions for machine learning.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Imitating Interactive Intelligence (2020, background): “The most direct approach to imitation learning, known as behavioural cloning (BC) (Pomerleau, 1989; Osa et al., 2018), frames the problem of copying behaviour as a supervised sequence prediction problem (Graves, 2013).”
Adversarial Training for High-Stakes Reliability (2022, background): “Future work could explore ways to generate synthetic examples, such as imitation learning on human examples [53] or better methods of using reinforcement learning to ﬁne-tune automated adversaries [37].”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, background): “rabi et al., 2018; Le et al., 2017; 2018) and many other areas such as playing games (Mnih et al., 2013). Behavioral cloning (Bain & Sommut, 1999) is one of the underlying central approaches. See Osa et al. (2018) for a thorough survey and more references therein. If we are allowed to access an ”
Implicit Behavioral Cloning (2021, background): “In addition to behavioral cloning (BC) [1], the machine learning and robotics communities have explored many additional approaches in imitation learning [51, 52, 53], often in ways that need additional information.”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “demonstration by a human-expert), then behavioral cloning should be utilized for learning from D (Osa et al., 2018).”
Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance (2019, background): “Comprehensive surveys on LfD [1], [18]–[20] do exist, but they mainly focus on summarizing existing LfD approaches, proposing taxonomy, and reporting challenges associated with employing LfD approaches in practice.”
Simitate: A Hybrid Imitation Learning Benchmark (2019, neutral): “The importance of such a metric has already been highlighted in 2009 [16] and again in 2018 [17].”
A Survey of Reinforcement Learning Informed by Natural Language (2019, background): “See [ Sutton and Barto, 2018 ] for a comprehensive introduction to RL and to [ Arulkumaran et al. , 2017 ] and [ Osa et al. , 2018 ] for reviews on recent algorithmic developments.”
Reward-Conditioned Policies (2019, background): “In contrast, supervised learning is comparatively well understood, and even imitation learning methods can often provide a much simpler approach to learning effective policies when demonstration data is available [3, 27, 33].”
Imitation Learning as f-Divergence Minimization (2019, neutral): “Imitation learning (IL) has a long-standing history in robotics as a tool to program desired skills and behavior in autonomous machines [7–10].”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, background): “While there is a large body of research on learning from demonstrations (Argall et al., 2009; Gao et al., 2012; Osa et al., 2018; Arora & Doshi, 2018), most work assumes access to action labels, while we learn only from observations.”
XIRL: Cross-embodiment Inverse Reinforcement Learning (2021, background): “Traditional formulations of imitation learning [8, 1, 9] assume access to a corpus of expert demonstration 326 data which includes both state and action trajectories of the expert policy.”
Learning From Demonstration in the Wild (2018, methodology): “While LfD has succeeded in a wide range of problems [1], [2], [3], nearly all methods rely on either artificially generated demonstrations (e.”
Synthetic Data for Deep Learning (2019, background): “, a more than 99% reduction in required real world input. Another direction where synthetic data might be useful for learning control is to generate synthetic behaviours to improve imitation learning [433]. Bansal et al. [35] discuss the insucient data problem in imitation learning: for learning to ”
Recent Advances in Robot Learning from Demonstration (2020, background): “(11) presented a more recent account of the same topic that focused on a more algorithmic perspective.”
Learning to Optimize Join Queries With Deep Reinforcement Learning (2018, background): “In the context of sequential planning, this problem seing is oen called Imitation Learning [42]; where one imitates an oracle as best as possible.”
Human motion trajectory prediction: a survey (2019, background): “In this section we discuss algorithms that estimate the reward function of agents (or directly a policy) from observations, using statistical and imitation learning techniques (for a survey on imitation learning techniques applied to robotic systems we refer the reader to (Osa et al. 2018)).”
Bayesian Robust Optimization for Imitation Learning (2020, background): “Imitation learning [37] aims to train an agent without hand-specifying a reward function by providing demonstrations.”
Learning from Interventions: Human-robot interaction as both explicit and implicit feedback (2020, background): “Traditional imitation learning is known to require a large amount (quadratic in trajectory length) of expert demonstration trajectories in order to achieve expert performance [8, 9] because learner policies invariably make mistakes and deviate from expert, inducing a significantly different distribu”