Imitation Learning with Demonstrations and Shaping Rewards
By Kshitij Judah, Alan Fern, Prasad Tadepalli, Robby Goetschalckx. 2014
44 citations recorded by the source corpus.
Imitation Learning (IL) is a popular approach for teaching behavior policies to agents by demonstrating the desired target policy. While the approach has lead to many successes, IL often requires a large set of demonstrations to achieve robust learning, which can be expensive for the teacher. In this paper, we consider a novel approach to improve the learning efficiency of IL by providing a shaping reward function in addition to the usual demonstrations. Shaping rewards are numeric functions of states (and possibly actions) that are generally easily specified, and capture general principles of desired behavior, without necessarily completely specifying the behavior. Shaping rewards have been used extensively in reinforcement learning, but have been seldom considered for IL, though they are often easy to specify. Our main contribution is to propose an IL approach that learns from both shaping rewards and demonstrations. We demonstrate the effectiveness of the approach across several IL problems, even when the shaping reward is not fully consistent with the demonstrations.
Subfields: Artificial Intelligence, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Peter Stone
- Matthew E. Taylor
- Bei Peng
- Phillip Odom
- Michael L. Littman
- James MacGlashan
- Sergey Levine
- Anthony Francis
In Control and Systems Engineering
- Matthew E. Taylor
- Michael L. Littman
- Aleksandra Faust
- David L. Roberts
- Sriraam Natarajan
- Bei Peng
- James MacGlashan
- Robert Tyler Loftin
Citations in the corpus
- Learning Navigation Behaviors End-to-End With AutoRL (2018, background): “Reward shaping [21], [36] addresses this problem by introducing a proxy reward function that is less sparse than the true objective.”
- Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning (2015, background): “In many cases [12,17] feedback is treated as being representative of some numeric reward or value function associated with the underlying task.”
- Human-Guided Learning for Probabilistic Logic Models (2018, methodology): “We next consider a sequential learning setting using the driving simulator extended from Judah et al. (2014).”