Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications
By Daniel S. Brown, Scott Niekum. 2018
85 citations recorded by the source corpus.
Inverse reinforcement learning (IRL) infers a reward function from demonstrations, allowing for policy improvement and generalization. However, despite much recent interest in IRL, little work has been done to understand the minimum set of demonstrations needed to teach a specific sequential decisionmaking task. We formalize the problem of finding maximally informative demonstrations for IRL as a machine teaching problem where the goal is to find the minimum number of demonstrations needed to specify the reward equivalence class of the demonstrator. We extend previous work on algorithmic teaching for sequential decision-making tasks by showing a reduction to the set cover problem which enables an efficient approximation algorithm for determining the set of maximallyinformative demonstrations. We apply our proposed machine teaching algorithm to two novel applications: providing a lower bound on the number of queries needed to learn a policy using active IRL and developing a novel IRL algorithm that can learn more efficiently from informative demonstrations than a standard IRL approach.
Subfields: Artificial Intelligence, Computer Networks and Communications
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yuke Zhu
- Li Fei-Fei
- Silvio Savarese
- Ajay Mandlekar
- Roberto Mart'in-Mart'in
- Sergey Levine
- Danfei Xu
- Scott Niekum
In Computer Networks and Communications
- George Dimitri Konidaris
- Oliver Kroemer
- Silvio Savarese
- Danfei Xu
- Yuke Zhu
- Scott Niekum
- Li Fei-Fei
- Ajay Mandlekar
Citations in the corpus
- Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “Other paradigms for human-in-the-loop policy learning include collaboration [14, 15], teaching the robot through informative sample selection [7, 9, 17], and leveraging physical kinesthetic corrections [5, 6].”
- A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “assumptions about the informativeness of the demonstrator [45, 187].”