Apprenticeship Learning using Inverse Reinforcement Learning and Gradient Methods

By Gergely Neu, Csaba Szepesvari. 2007

251 citations recorded by the source corpus.

In this paper we propose a novel gradient algorithm to learn a policy from an expert's observed behavior assuming that the expert behaves optimally with respect to some unknown reward function of a Markovian Decision Problem. The algorithm's aim is to find a reward function such that the resulting optimal policy matches well the expert's observed behavior. The main difficulty is that the mapping from the parameters to policies is both nonsmooth and highly redundant. Resorting to sub differentials solves the first difficulty, while the second one is overcome by computing natural gradients. We tested the proposed method in two artificial domains and found it to be more reliable and efficient than some previous methods.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, neutral): “However, as shown in Finn et al. (2017b) and similar to the setup in Neu & Szepesv´ari (2012), we actually have considerable ﬂexibility here to make the inner and outer objectives different.”
Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “In Ng et al. (2000); Boularias et al. (2011); Neu and Szepesvári (2012); Wulfmeier et al. (2015), inverse reinforcement learning problems have been considered, where the objective is to extract (unknown) reward function from the trajectories given by an oracle or expert.”
Value Iteration Networks (2016, background): “VINs are fundamentally different than inverse RL methods [22], where transitions are required to be known.”
Learning with a Strong Adversary (2015, background): “Thus, by Proposition 2 of (Neu & Szepesv ´ ari, 2012), ( u ∗ , v 0 ) also belongs to the subderivative of L .”
Maximum Entropy Deep Inverse Reinforcement Learning (2015, background): “A number of approaches have been proposed to tackle the IRL problem (see, for example, (Abbeel & Ng, 2004), (Neu & Szepesv´ari, 2012), (Ratliff et al., 2006), (Syed & Schapire, 2007)).”
Inverse Reinforcement Learning in Swarm Systems (2016, methodology): “Depending on the single-agent IRL framework in use, this involves an algorithm-speciﬁc optimization procedure, e.g. in the form of a quadratic program [2, 20] or a gradient-based optimization [19, 35].”
Apprenticeship Learning About Multiple Intentions (2011, background): “Policy matching (Neu & Szepesv´ari, 2007) tries to make the actions taken by its policy as close as possible to those observed from the expert.”
Relative Entropy Inverse Reinforcement Learning (2011, methodology): “, 2009), the natural gradient approach (Neu and Szepesvari, 2007), and the game-theoretic approach (Syed and Schapire, 2008).”
Learning grounded finite-state representations from unstructured demonstrations (2015, methodology): “IRL techniques typically model the problem as a Markov decision process (MDP) and require an accurate model of the environment (Abbeel and Ng, 2004; Ramachandran and Amir, 2007; Neu and Szepesvári, 2007), but some recent methods have been proposed to circumvent this requirement by creating local con”
Learning to search: Functional gradient techniques for imitation learning (2009, methodology): “Importantly, using the modified correlation criterion removes the feature scaling problem discussed in Neu and Szepesvari (2007 ) r eferenced in Sect.”
Active Learning for Reward Estimation in Inverse Reinforcement Learning (2009, methodology): “This problem has since been addressed in several other works [4, 5, 6, 7, 8].”
Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamics (2016, methodology): “(11), which shares similarities with the approach in (Neu and Szepesvári, 2007).”
Nonparametric Bayesian Inverse Reinforcement Learning for Multiple Reward Functions (2012, background): “Under this assumption, a number of studies on IRL have appeared in the literature [2] [3] [4] [5].”
A New Softmax Operator for Reinforcement Learning (2016, methodology): “Algorithms for inverse reinforcement learning (IRL), the problem of inferring reward functions from observed behavior (Ng & Russell, 2000), frequently use a Boltzmann operator to avoid assigning zero probability to non-optimal actions and hence assessing an observed sequence as impossible.”
Abstraction Levels for Robotic Imitation: Overview and Computational Approaches (2010, methodology): “We adopt the method described in [89], which is a basic variation of the Bayesian inverse reinforcement learning (BIRL) algorithm in [116], but the same problem could be tackled using other IRL methods from the literature (see, for example, [102, 136]).”
Learning to search: structured prediction techniques for imitation learning (2009, methodology): “In both (Neu & Szepesvari, 2007) and (Ramachandran & Amir, 2007), the authors utilize the Gibbs model as a smooth approximation to the hypothesized policy in order to utilize a variety of loss functions under IOC.”
Convex Optimization: Algorithms and Complexity (2015, unknown): “Both reward learning (direct) and apprenticeship learning (indirect) views of this problem have been studied in the last decade (e.g., [Ng and Russell, 2000, Abbeel and Ng, 2004, Ratliff et al., 2006, Neu and Szepesvári, 2007, Ziebart et al., 2008, Syed and Schapire, 2008]).”
Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion (2007, unknown): “While most reinforcement learning algorithms operate under the Markov decision process (MDP) formalism (where the reward function is typically assumed to be given a priori), past work [1, 13, 11] has noted that often the reward function itself is difficult to specify by hand, since it must quantify ”