Learning from Demonstration for Shaping through Inverse Reinforcement Learning

By Halit Bener Suay, Tim Brys, Matthew E. Taylor, Sonia Chernova. 2016

81 citations recorded by the source corpus.

Model-free episodic reinforcement learning problems define the environment reward with functions that often provide only sparse information throughout the task. Consequently, agents are not given enough feedback about the fitness of their actions until the task ends with success or failure. Previous work addresses this problem with reward shaping. In this paper we introduce a novel approach to improve model-free reinforcement learning agents' performance with a three step approach. Specifically, we collect demonstration data, use the data to recover a linear function using inverse reinforcement learning and we use the recovered function for potential-based reward shaping. Our approach is model-free and scalable to high dimensional domains. To show the scalability of our approach we present two sets of experiments in a two dimensional Maze domain, and the 27 dimensional Mario AI domain. We compare the performance of our algorithm to previously introduced reinforcement learning from demonstration algorithms. Our experiments show that our approach outperforms the state-of-the-art in cumulative reward, learning rate and asymptotic performance.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep Q-learning From Demonstrations (2017, methodology): “Follow-ups to this work showed how expert advice or demonstrations can be used to shape rewards in the RL problem (Brys et al. 2015; Suay et al. 2016).”
Learning from Demonstrations for Real World Reinforcement Learning (2017, background): “Follow-ups to this work showed how expert advice or demonstrations can be used to shape rewards in the RL problem (Brys et al., 2015; Suay et al., 2016).”