Generative Adversarial Imitation Learning
By Jonathan Ho, Stefano Ermon. 2016
3,722 citations recorded by the source corpus.
Consider learning a policy from example expert behavior, without interaction with the expert or access to reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Mohit Bansal
- Chelsea Finn
- Ruslan Salakhutdinov
- Quoc V. Le
- Jianfeng Gao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
- Andrea Vedaldi
In Statistical and Nonlinear Physics
- Sergey Levine
- Stefano Ermon
- Pieter Abbeel
- Yang Song
- Chelsea Finn
- Yoshua Bengio
- Jiaming Song
- Arash Vahdat
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Yuxi Li
- Shixiang Shane Gu
- Abhishek Gupta
Citations in the corpus
- A General Language Assistant as a Laboratory for Alignment (2021, neutral): “The authors of [AAB21] compared GAIL [HE16] to conventional imitation learning in an RL context, and found in some cases that GAIL scaled significantly better with dataset size.”
- Deep Reinforcement Learning from Human Preferences (2017, methodology): “, 2016), imitation learning (Ho and Ermon, 2016; Stadie et al., 2017), semi-supervised skill generalization (Finn et al.”
- Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Generative Adversarial Networks (GANs) [3] are state-of-the-art models for a large variety of tasks such as image generation [4], semi-supervised learning [5], image editing [6], image translation [7], and imitation learning [8].”
- Generative Modeling by Estimating Gradients of the Data Distribution (2019, background): “To list a few, they have been used to generate high-fidelity images [26, 6], synthesize realistic speech and music fragments [58], improve the performance of semi-supervised learning [28, 10], detect adversarial examples and other anomalous data [54], imitation learning [22], and explore promising s”
- Spectral Normalization for Generative Adversarial Networks (2018, methodology): “…adversarial networks (GANs) (Goodfellow et al., 2014) have been enjoying considerable success as a framework of generative models in recent years, and it has been applied to numerous types of tasks and datasets (Radford et al., 2016; Salimans et al., 2016; Ho & Ermon, 2016; Li et al., 2017).”
- One-Shot Imitation Learning (2017, background): “…Pomerleau (1989); Ross et al. (2011)); and inverse reinforcement learning (Ng & Russell, 2000), where a reward function (Abbeel & Ng, 2004; Ziebart et al., 2008; Levine et al., 2011; Finn et al., 2016; Ho & Ermon, 2016) is estimated that explains the demonstrations as (near) optimal behavior.”
- Scalable agent alignment via reward modeling: a research direction (2018, methodology): “Reinforcement learning from a learned reward function has been successful in gridworlds (Bahdanau et al., 2018), Atari games (Christiano et al., 2017; Ibarz et al., 2018), and continuous motor control tasks (Ho & Ermon, 2016; Christiano et al., 2017).”
- Reward learning from human preferences and demonstrations in Atari (2018, methodology): “When learning from demonstrations, a policy model is trained to imitate a human demonstrator on the task (Ho and Ermon, 2016; Hester et al., 2018).”
- Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “…Inverse Reinforcement Learning (IRL; Ng & Russell, 2000; Ziebart et al., 2008) methods in general, and Generative Adversarial Imitation Learning (Ho & Ermon, 2016) in particular, have yielded some success in jointly learning reward functions from expert data and training policies from learned…”
- Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “Ho and Ermon (2016) propose a generative adversarial network based imitation learning algorithm that computes the optimal policy directly from the trajectories of expert.”
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “Recent work has also drawn connections between generative adversarial networks (GANs) (Goodfellow et al., 2014) and maximum entropy inverse reinforcement learning (Finn et al., 2016b,a; Fu et al., 2018; Ho and Ermon, 2016).”
- Imitating Interactive Intelligence (2020, methodology): “In addition to training the agent via a supervised method such as behavioural cloning we also used a form of inverse reinforcement learning, specifically Generative Adversarial Imitation Learning (GAIL) (Ho and Ermon, 2016).”
- Shaking the foundations: delusions in sequence models for interaction and control (2021, background): “This is a simple question with far-reaching implications on foundation models [4], language agents [5], behavior cloning [10], inverse reinforcement learning [1, 13], goal-/return-conditioned policies [6, 11, 22]), and so forth.”
- Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Incorporating expert trajectories in PCL is relatively trivial compared to the specialized methods developed for other policy based algorithms [1, 12].”
- NIPS 2016 Tutorial: Generative Adversarial Networks (2016, neutral): “, 2016a), and have applied GANs to imitation learning (Ho and Ermon, 2016).”
- Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “For instance, [70] embedded GANs into imitation learning; [61] combined GANs with policy gradient; [71] described the relationship between GANs and actor-critic algorithms, etc.”
- Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “…known to fail when exposed to noisy or sub-optimal trajectories (Ross et al., 2011), and more sophisticated approaches are generally unable to surpass the performance of the demonstrator without further interactions with the environment (Ziebart et al., 2008; Ho & Ermon, 2016; Hester et al., 2017).”
- Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Generative Adversarial Imitation Learning (GAIL) [73] introduces a way to avoid this expensive inner loop.”
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “Inspired by Generative Adversarial Networks (GANs) (Good-fellow et al., 2014), Ho and Ermon (2016) propose Generative Adversarial Imitation Learning (GAIL), which surpasses the intermediate step of learning a reward function and is capable of directly learning a policy from expert demonstrations.”
- Deep Q-learning From Demonstrations (2017, background): “Another popular paradigm is to setup a zero-sum game where the learner chooses a policy and the adversary chooses a reward function (Syed and Schapire 2007; Syed, Bowling, and Schapire 2008; Ho and Ermon 2016).”