A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress

By Saurabh Arora, Prashant Doshi. Artificial Intelligence, 2018

794 citations recorded by the source corpus.

Inverse reinforcement learning is the problem of inferring the reward function of an observed agent, given its policy or behavior. Researchers perceive IRL both as a problem and as a class of methods. By categorically surveying the current literature in IRL, this article serves as a reference for researchers and practitioners in machine learning to understand the challenges of IRL and select the approaches best suited for the problem on hand. The survey formally introduces the IRL problem along with its central challenges which include accurate inference, generalizability, correctness of prior knowledge, and growth in solution complexity with problem size. The article elaborates how the current methods mitigate these challenges. We further discuss the extensions of traditional IRL methods: (i) inaccurate and incomplete perception, (ii) incomplete model, (iii) multiple rewards, and (iv) non-linear reward functions. This discussion concludes with some broad advances in the research area and currently open research questions.

Subfields: Artificial Intelligence, Management Information Systems

View Paper: https://www.semanticscholar.org/paper/9d4d8509f6da094a7c31e063f307e0e8592db27f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

Citations in the corpus

A review of cooperative multi-agent deep reinforcement learning (2019, background): “While this area attained significant attention in the singleagent RL problems [9], there is no remarkable contribution regarding the inverse RL for MARL.”
Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications (2018, methodology): “This property is true of all standard IRL methods (Gao et al. 2012; Arora and Doshi 2018).”
Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review (2022, background): “…to varying degrees in this review, but detailed reviews for each are provided in the following papers (Game ToM in Yoshida et al., 2008; Observational Reinforcement Learning in Albrecht and Stone, 2018; Inverse Reinforcement Learning in Arora and Doshi, 2021 and Bayesian ToM in Baker et al., 2011).”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, methodology): “Inverse reinforcement learning (IRL) methods are another way of performing imitation learning (Arora & Doshi, 2018), where the learning agent seeks to achieve better sample efficiency and generalization by learning a reward function which is then optimized to obtain a policy.”
A Survey of Human-in-the-loop for Machine Learning (2021, methodology): “Besides, applying inverse reinforcement learning also seems to be a feasible and practical solution [141].”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, background): “Rather than directly learn to mimic the demonstrator, inverse reinforcement learning (IRL) (Gao et al., 2012; Arora & Doshi, 2018) seeks to ﬁnd a reward function that models the intention of the demonstrator, thereby allowing generalization to states that were unvisited during demonstration.”