Learning Robust Rewards with Adversarial Inverse Reinforcement Learning

By Justin Fu, Katie Luo, Sergey Levine. 2017

869 citations recorded by the source corpus.

Reinforcement learning provides a powerful and general framework for decision making and control, but its application in practice is often hindered by the need for extensive feature and reward engineering. Deep reinforcement learning methods can remove the need for explicit engineering of policy or value features, but still require a manually specified reward function. Inverse reinforcement learning holds the promise of automatic reward acquisition, but has proven exceptionally difficult to apply to large, high-dimensional problems with unknown dynamics. In this work, we propose adverserial inverse reinforcement learning (AIRL), a practical and scalable inverse reinforcement learning algorithm based on an adversarial reward learning formulation. We demonstrate that AIRL is able to recover reward functions that are robust to changes in dynamics, enabling us to learn policies even under significant variation in the environment seen during training. Our experiments show that AIRL greatly outperforms prior methods in these transfer settings.

Subfields: Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “A recent extension of this framework also provides for an effective inverse reinforcement learning algorithm in a model-free deep RL context, as well as a mechanism for recovering robust and transferable rewards in ambiguous settings (Fu et al., 2018).”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “Many existing imitation learning works (Ho and Ermon, 2016; Fu et al., 2017; Kostrikov et al., 2018; Ke et al., 2019; Ghasemipour et al., 2019) aim to exactly optimize this f -divergence objective.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, background): “Inverse reinforcement learning (e.g., see [Abbeel and Ng, 2004, Ng et al., 2000, Ziebart et al., 2008, Finn et al., 2016a,b, Fu et al., 2017]) is another important and successful line of ideas for imitation learning.”
The MAGICAL Benchmark for Robust Imitation (2020, methodology): “[17] trained an algorithm on demonstrations from the standard “Ant” task from Gym, then tested on a variant of the task where two of the creature’s”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, methodology): “The discriminator in AIRL is parameterized by ψ and given by the following equation: Dψ(s,a) = exp(fψ(s,a)) exp(fψ(s,a)) + π(a|s) .”
Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning (2018, methodology): “Similarly, the Adversarial Inverse Reinforcement Learning (AIRL) algorithm (Fu et al., 2017) makes use of a modiﬁed GAIL discriminator to recover a reward function that can be used to perform Inverse Reinforcement Learning (IRL) (Abbeel & Ng, 2004) and who‘s subsequent dense reward is robust to…”
Imitation Learning as f-Divergence Minimization (2019, methodology): “AIRL [34], EAIRL [35] uses this to recover rewards.”
Imitation Learning via Off-Policy Distribution Matching (2019, background): “Distribution matching via adversarial learning, or Adversarial Imitation Learning (AIL), has recently become a popular approach for imitation learning (Ho & Ermon, 2016; Fu et al., 2017; Ke et al., 2019; Kostrikov et al., 2019).”
Task-Relevant Adversarial Imitation Learning (2019, background): “Although many variants are introduced in the literature [Li et al., 2017, Fu et al., 2018, Merel et al., 2017, Zhu et al., 2018, Baram et al., 2017], making GAIL work for high-dimensional input spaces, particularly raw pixels, remains a challenge.”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, background): “Most deep learning-based methods for reward learning require access to demonstrator actions and do not scale to high-dimensional tasks such as video games (e.g. Atari) (Ho & Ermon, 2016; Finn et al., 2016; Fu et al., 2017; Qureshi & Yip, 2018).”
Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games (2021, methodology): “However, notice that we are maximizing a f -divergence objective of occupancy measures, while occupancy measure matching algorithms in imitation learning try to minimize the same objective [12, 10, 9].”
Efficient Exploration via State Marginal Matching (2019, methodology): “Many of the underlying ingredients of our method, such as adversarial games and density estimation, have seen recent progress in imitation learning (Ziebart et al., 2008; Ho & Ermon, 2016; Finn et al., 2016; Fu et al., 2017).”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, background): “Reward functions are often parameterized as a linear combination of known features (Abbeel & Ng, 2004; Ziebart et al., 2008; Sadigh et al., 2017) or as a deep neural network (Ho & Ermon, 2016; Fu et al., 2017).”
Imitation Learning from Video by Leveraging Proprioception (2019, background): “, 2019] and improving reward representation [Fu et al., 2018; Qureshi et al., 2019].”
Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning (2020, background): “The state-of-the-art approaches to this problem are based on a GAN framework where π θ is treated as a generator, and a discriminator D w : S × A → (0 , 1) , doubling as a reward function, is learned to discriminate between state-action pairs generated by π E and π θ [16], [17], [20].”
Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, background): “Another interesting RL variant is “inverse RL” [37], [38], in which the state-action sequence of an (expert) agent is given, and the task is to infer the reward function that this agent seeks to maximize.”