Model-Free Imitation Learning with Policy Optimization

By Jonathan Ho, Jayesh K. Gupta, Stefano Ermon. 2016

157 citations recorded by the source corpus.

In imitation learning, an agent learns how to behave in an environment with an unknown cost function by mimicking expert demonstrations. Existing imitation learning algorithms typically involve solving a sequence of planning or reinforcement learning problems. Such algorithms are therefore not directly applicable to large, high-dimensional environments, and their performance can significantly degrade if the planning problems are not solved to optimality. Under the apprenticeship learning formalism, we develop alternative model-free algorithms for finding a parameterized stochastic policy that performs at least as well as an expert policy on an unknown cost function, based on sample trajectories from the expert. Our approach, based on policy gradients, scales to large continuous environments with guaranteed convergence to local minima.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “, 2017), and modified policy gradient approaches (Ho et al., 2016; Sun et al., 2017; Cheng et al., 2018; Sun et al., 2018).”
Implicit Behavioral Cloning (2021, background): “Although considerable research has been devoted to 22 developing new imitation learning methods [7, 8, 9] to address BC’s known limitations, here we investigate 23 a fundamental design decision that has largely been overlooked: the form of the policy itself.”
A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models (2016, methodology): “[10, 9] previously presented a GAN-like algorithm f or imitation learning, where the goal is to recover a policy that matches the expert demonstration s.”
Imitation Learning as f-Divergence Minimization (2019, background): “The degeneracy issues of IRL can be alleviated by a game theoretic framework where an adversary selects a reward function and the learner must compete to do as well as the expert [29, 30].”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “Scaling IRL algorithms to larger environments has posed a signiﬁcant challenge, despite their success in producing policies mirroring expert behavior [64], [71], [72].”
Generative Adversarial Imitation Learning (2016, methodology): “[10] using the cost function class Clinear (10) of Abbeel and Ng [1]”
BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning (2019, background): “…problem (Schaal, 1999; Argall et al., 2009; Hussein et al., 2017) and also continues to be a highly active area of research (Kim et al., 2013; Piot et al., 2014; Chemali & Lazaric, 2015; Hester et al., 2018; Ho et al., 2016; Sun et al., 2017; 2018; Cheng et al., 2018; Gao et al., 2018).”
Stabilizing Adversarial Nets With Prediction Methods (2017, unknown): “…Wang & Gupta, 2016; Isola et al., 2017), domain adaptation (Taigman et al., 2017; Tzeng et al., 2017; Ganin & Lempitsky, 2015), imitation learning (Ho et al., 2016), privacy (Edwards & Storkey, 2016; Abadi & Andersen, 2016), fair representation (Mathieu et al., 2016; Edwards & Storkey, 2016), etc.”
Deep Reinforcement Learning: An Overview (2017, background): “Behavioral cloning, or apprenticeship learning, or learning from demonstration, is formulated as a supervised learning problem to map state-action pairs from expert trajectories to policy, without learning the reward function (Ho et al., 2016; Ho and Ermon, 2016).”
Inferring The Latent Structure of Human Decision-Making from Raw Visual Inputs (2017, background): “AL, on the other hand, has the advantage of learning a reward function that can be used to score entire trajectories (Abbeel & Ng, 2004; Syed et al., 2008; Ho et al., 2016), but is typically expensive to run because it requires solving a reinforcement learning (RL) problem inside a learning loop.”
Cooperative Multi-agent Control Using Deep Reinforcement Learning (2017, unknown): “The extension is based on the single agent waterworld domain used by [15] .”
Imitation Learning via Kernel Mean Embedding (2018, background): “Ho, Gupta, and Ermon (2016) presented a gradient-based stochastic optimization approach, where the parameterized policy (typically a neural network) πθ is found by alternating between computing the cost function c∗ that attains maximum in ψ∗C with fixed πθ, and then improving πθ by gradient descent…”
State Aware Imitation Learning (2017, background): “Recently, Ho et al.[11] introduced an approach that aims to ﬁnd a policy that implicitly maximizes a linear reward function but without the need to explicitly represent such a reward function.”
Imitating driver behavior with generative adversarial networks (2017, background): “Instead, recent work has attempted to imitate expert behavior through direct policy optimization, without first learning a cost function [16], [17].”