A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models

By Chelsea Finn, Paul Christiano, Pieter Abbeel, Sergey Levine. 2016

382 citations recorded by the source corpus.

Generative adversarial networks (GANs) are a recently proposed class of generative models in which a generator is trained to optimize a cost function that is being simultaneously learned by a discriminator. While the idea of learning cost functions is relatively new to the field of generative modeling, learning costs has long been studied in control and reinforcement learning (RL) domains, typically for imitation learning from demonstrations. In these fields, learning cost function underlying observed behavior is known as inverse reinforcement learning (IRL) or inverse optimal control. While at first the connection between cost learning in RL and cost learning in generative modeling may appear to be a superficial one, we show in this paper that certain IRL methods are in fact mathematically equivalent to GANs. In particular, we demonstrate an equivalence between a sample-based algorithm for maximum entropy IRL and a GAN in which the generator's density can be evaluated and is provided as an additional input to the discriminator. Interestingly, maximum entropy IRL is a special case of an energy-based model. We discuss the interpretation of GANs as an algorithm for training energy-based models, and relate this interpretation to other recent work that seeks to connect GANs and EBMs. By formally highlighting the connection between GANs, IRL, and EBMs, we hope that researchers in all three communities can better identify and apply transferable ideas from one domain to another, particularly for developing more stable and scalable algorithms: a major challenge in all three domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Generative Models

Citations in the corpus

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “The connection between GANs and EBMs has been studied by many authors (Zhao et al., 2016; Finn et al., 2016).”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “On the other end of the spectrum of deep generative models lies the family of energy-based models (EBMs) [42, 54, 29, 77, 73, 13, 38, 55, 9, 11, 16, 18, 6].”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “extensions to approximate inference via the Laplace approximation (under known dynamics) (Levine and Koltun, 2012; Draganet al., 2013) and approximate reinforcement learning (under unknown dynamics) (Finn et al., 2016b; Fu et al., 2018). Aside from inferring reward functions from demonstrations for ”
Imitating Interactive Intelligence (2020, methodology): “By contrast, inverse reinforcement learning (IRL) algorithms (Ziebart, 2010; Finn et al., 2016) attempt to infer the reward function underlying the intentions of the demonstrator (e.g., which states it prefers), and optimise the policy itself using reinforcement learning to pursue this reward…”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, methodology): “Researchers have already identified connections between GANs and actor-critic methods (Pfau and Vinyals, 2016), inverse reinforcement learning (Finn et al., 2016a), and have applied GANs to imitation learning (Ho and Ermon, 2016).”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, background): “Inverse reinforcement learning (e.g., see [Abbeel and Ng, 2004, Ng et al., 2000, Ziebart et al., 2008, Finn et al., 2016a,b, Fu et al., 2017]) is another important and successful line of ideas for imitation learning.”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, background): “, learning a return-maximizing policy – other works have established connections between energy-based models and inverse RL or imitation learning (Finn et al., 2016).”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “While in principle this would require importance sampling if using off-policy data from the replay buffer R, prior work has observed that adversarial IRL can drop the importance weights both in theory [10] and in practice [22].”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “policy in [Finn et al., 2016b] in order to learn a complex task with a monolithic policy.”
Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (2017, methodology): “Our experimental evaluation demonstrates that AIRL outperforms prior IRL meth-ods (Finn et al., 2016b) on continuous, high-dimensional tasks with unknown dynamics by a wide margin.”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, background): “EBMs [27, 33, 21, 47, 10, 43, 6, 7, 25, 35, 5, 11, 4, 8, 2, 12] are an appealing class of probabilistic models, which can be viewed as generative versions of discriminators [18, 26, 28, 12], yet can be learned from unlabeled data.”
Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning (2018, methodology): “Operating in the Maximum Entropy IRL formulation (Ziebart et al., 2008), Finn et al. (2016b) introduce an iterative-sampling based estimator for the partition function, deriving an algorithm for recovering non-linear reward functions in high-dimensional state and action spaces.”
PacGAN: The Power of Two Samples in Generative Adversarial Networks (2017, neutral): “…the visual quality of an image is the inception score , which describes how precisely a classiﬁer can classify generated images, thereby implicitly capturing some measure of image quality [60]; this has become a de facto evaluation technique for GANs trained on realistic data [60, 17, 20, 49].”
Implicit Generation and Generalization in Energy-Based Models (2019, background): “…et al., 2016; Haarnoja et al., 2017) pro-pose learning a separate network to generate samples, which makes these methods closely connected to generative adversarial networks as shown by (Finn et al., 2016), but these methods do not have the advantages of implicit sampling noted in the introduction.”
Unsupervised Perceptual Rewards for Imitation Learning (2016, methodology): “Prior work has addressed this using dynamic programming to exactly compute Z in small, discrete systems (Ziebart et al., 2008), or by using sampling to estimate Z for large, continuous domains (Kalakrishnan et al., 2010; Boularias et al., 2011; Finn et al., 2016b).”
A Divergence Minimization Perspective on Imitation Learning Methods (2019, methodology): “Recent scalable approaches to Max-Ent IRL [2, 1, 12], motivated by adversarial approaches to generative modeling [33], demonstrate additional connections to distribution matching.”
Imitation Learning as f-Divergence Minimization (2019, background): “Hence IRL can be connected to min-max formulations [31] like GANs [32].”
Task-Relevant Adversarial Imitation Learning (2019, methodology): “Recently, Finn et al. [2016b] approached continuous robotic control problems with success by applying Maximum Entropy IRL algorithms which are very closely related to GAIL [Finn et al., 2016a] and have similar drawbacks.”
Generative adversarial networks: introduction and outlook (2017, background): “Some researchers connect GANs with imitation learning [51], [52] or Actor-critic based methods [53].”