Maximum Entropy Inverse Reinforcement Learning

By Brian D. Ziebart, Andrew L. Maas, J. Bagnell, Anind K. Dey. 2008

3,476 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, neutral): “The energy function in the energy-based model can be interpreted as the cost function for optimal control, and the dynamic generator model consists of the policy and the dynamics [65].”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “While the learned short-run MCMC can be used for synthesis, the above learning scheme can be generalized to tasks such as image inpainting, super-resolution, style transfer, or inverse optimal control [52, 1] etc.”
One-Shot Imitation Learning (2017, background): “, [41, 44]); and inverse reinforcement learning [37], where a reward function [1, 66, 29, 18, 22] is estimated that explains the demonstrations as (near) optimal behavior.”
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, background): “[65]); for example, G-learning penalizes KL-divergence from a simple uniform distribution in order to cope with overestimation of Q-values [14].”
Reward learning from human preferences and demonstrations in Atari (2018, methodology): “In those cases we can rely on demonstrations from human experts (inverse reinforcement learning, Ng and Russell, 2000; Ziebart et al., 2008), policy feedback (Knox and Stone, 2009; Warnell et al., 2017), or trajectory preferences (Wilson et al., 2012; Christiano et al., 2017).”
Learning to Understand Goal Specifications by Modelling Reward (2018, background): “When there is a single implicit task, Inverse Reinforcement Learning (IRL; Ng & Russell, 2000; Ziebart et al., 2008) methods in general, and Generative Adversarial Imitation Learning (Ho & Ermon, 2016) in particular, have yielded some success in jointly learning reward functions from expert data…”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “Indeed, the use of the term “maximum entropy reinforcement learning” in this article is based on the work of Ziebart and colleagues, who proposed the maximum entropy inverse reinforcement learning algorithm (Ziebart et al., 2008) for inferring reward functions and modeling human behavior.”
Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, background): “There is also a connection between this work and Maximum Entropy Inverse RL (Ziebart et al., 2008), which can be seen as KL control with a flat, improper prior.”
Learning Trajectory Preferences for Manipulators via Iterative Improvement (2013, background): “Some later works in LfD provided ways for handling noisy demonstrations, under the assumption that demonstrations are either near optimal [39] or locally optimal [22].”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “…objective for the inﬁnite horizon discounted case is more involved (Thomas, 2014) and is deferred to Appendix A. Prior methods have proposed directly solving for the optimal Q-function, from which the optimal policy can be recovered (Ziebart et al., 2008; Fox et al., 2016; Haarnoja et al., 2017).”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “In this paper, we study the optimal state and action values for a softmax form of temporal consistency [48, 47, 7], which arises by augmenting the standard expected reward objective with a discounted entropy regularizer.”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “1 Optimization problems of this type have been explored in a number of prior works (Kappen, 2005; Todorov, 2007; Ziebart et al., 2008), which are covered in more detail in Section 4.”
Value Iteration Networks (2016, methodology): “We demonstrate that VINs can be effectively used within standard RL and IL algorithms in various problems, among which require visual perception, continuous control, and also natural language based decision making in the WebNav challenge [20].”
Generative Adversarial Imitation Learning (2016, methodology): “For the remainder of this paper, we will adopt and assume the existence of solutions of maximum causal entropy IRL [29, 30], which fits a cost function from a family of functions C with the optimization problem”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “ed to as Inverse RL (IRL) [53], [54], which has received an increasingly high interest by researchers in the past few years. Numerous IRL methods have been proposed, including the Maximum Entropy IRL [55], the Apprenticeship Learning [56], nonlinear representations of the reward function using Gauss”
Verifiable Reinforcement Learning via Policy Extraction (2018, methodology): “…policies trained using policy gradients (so no Q -function is available), we use the maximum entropy formulation of reinforcement learning to obtain Q values, i.e., Q ( s, a ) = log π ∗ ( s, a ) , where π ∗ ( s, a ) is the probability that the (stochastic) oracle takes action a in state s [37].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “Wulfmeier et al. (2015) extends Maximum Entropy IRL under the deep learning context, utilizing fully convolutional neural networks as the approximator for learning the reward function.”
Reinforcement learning in robotics: A survey (2013, methodology): “Ziebart et al. (2008) extended the technique developed by Abbeel and Ng (2004) by rendering the idea robust and probabilistic, enabling its effective use for both learning policies and predicting the behavior of sub-optimal agents.”
Self-Imitation Learning (2018, background): “The goal of entropy-regularized RL is to learn a stochastic policy which maximizes the entropy of the policy as well as the γ-discounted sum of rewards (Haarnoja et al., 2017; Ziebart et al., 2008):”