Maximum Entropy Deep Inverse Reinforcement Learning

By Markus Wulfmeier, Peter Ondruska, Ingmar Posner. 2015

458 citations recorded by the source corpus.

This paper presents a general framework for exploiting the representational capacity of neural networks to approximate complex, nonlinear reward functions in the context of solving the inverse reinforcement learning (IRL) problem. We show in this context that the Maximum Entropy paradigm for IRL lends itself naturally to the efficient training of deep architectures. At test time, the approach leads to a computational complexity independent of the number of demonstrations, which makes it especially well-suited for applications in life-long learning scenarios. Our approach achieves performance commensurate to the state-of-the-art on existing benchmarks while exceeding on an alternative benchmark based on highly varying reward structures. Finally, we extend the basic architecture - which is equivalent to a simplified subclass of Fully Convolutional Neural Networks (FCNNs) with width one - to include larger convolutions in order to eliminate dependency on precomputed spatial features and work on raw input representations.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “In Ng et al. (2000); Boularias et al. (2011); Neu and Szepesvári (2012); Wulfmeier et al. (2015), inverse reinforcement learning problems have been considered, where the objective is to extract (unknown) reward function from the trajectories given by an oracle or expert.”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “…in inverse reinforcement learning has studied settings where the reward function has a more complex, non-linear representation (Levine et al., 2011; Wulfmeier et al., 2015), and extensions to approximate inference via the Laplace approximation (under known dynamics) (Levine and Koltun, 2012;…”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “Wulfmeier et al. (2015) extends Maximum Entropy IRL under the deep learning context, utilizing fully convolutional neural networks as the approximator for learning the reward function.”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “Inverse reinforcement learning methods [47, 41, 11, 17, 12] seek to automate reward definition by learning a reward”
Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization (2016, background): “While neural network cost representations have previously been proposed in the literature (Wulfmeier et al., 2015), they have only been applied to small, synthetic domains.”
Third-Person Imitation Learning (2017, background): “The most closely related work to ours is by Finn et al. (2016); Ho & Ermon (2016); Wulfmeier et al. (2015), who also consider inverse reinforcement learning directly from raw sensory data.”
Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (2017, background): “Wulfmeier et al. (2015) also consider learning cost functions with neural networks, but only evaluate on simple domains where analytically solving the problem with value iteration is tractable.”
Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces (2017, methodology): “Deep learning has also been applied to several of the areas described above, including learning from demonstration (Hester et al. 2017) and inverse reinforcement learning (Wulfmeier, Ondruska, and Posner 2016).”
Concrete Problems for Autonomous Vehicle Safety: Advantages of Bayesian Deep Learning (2017, methodology): “Variants of IRL include Bayesian IRL which use Gaussian processes [Levine et al., 2011], as do deep IRL which use neural networks [Wulfmeier et al., 2015a].”
Imitation Learning as f-Divergence Minimization (2019, background): “Moment matching naturally leads to maximum entropy formulations [27] which has been used successfully in various applications [2, 28].”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “While there are several ways to perform imitation learning such as inverse reinforcement learning [30, 50, 10] and occupancy matching [19], we restrict our attention in this work to behavior cloning [35] given its stability and ease of use.”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, background): “Maximum entropy IRL does this matching, while further disambiguating inference by also maximizing the entropy of the resulting policy (Ziebart et al., 2008; Boularias et al., 2011; Wulfmeier et al., 2015; Finn et al., 2016).”
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning (2018, methodology): “In order to learn from high dimensional input, Wulfmeier et al. (2015) proposed applying fully convolutional networks (Shelhamer et al., 2017) to the MaxEnt IRL framework (Ziebart et al., 2008) for several navigation tasks (Wulfmeier et al., 2016a;b).”
Planning and Decision-Making for Autonomous Vehicles (2018, background): “The maximum-entropy deep IRL framework (122) exploits the expressive capacity of deep fully convolutional neural networks to represent the cost model underlying driving behaviors.”
Imitation Learning (2017, methodology): “This approach is extended to a deep-learning framework in [Wulfmeier et al. 2015].”
Reward (Mis)design for Autonomous Driving (2021, background): “Approaches that instead model reward via expressive representations like deep neural networks [58,10,16] could avoid this challenge.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “Inthe progression of the IRL methodologies, a notable development arose with the introduction of maximum entropy deep IRL by Wulfmeier et al. [63].”