RAIL: Risk-Averse Imitation Learning
By Anirban Santara, Abhishek Naik, Balaraman Ravindran, Dipankar Das, Dheevatsa Mudigere, Sasikanth Avancha, Bharat Kaul. 2017
21 citations recorded by the source corpus.
Imitation learning algorithms learn viable policies by imitating an expert's behavior when reward signals are not available. Generative Adversarial Imitation Learning (GAIL) is a state-of-the-art algorithm for learning policies when the expert's behavior is available as a fixed set of trajectories. We evaluate in terms of the expert's cost function and observe that the distribution of trajectory-costs is often more heavy-tailed for GAIL-agents than the expert at a number of benchmark continuous-control tasks. Thus, high-cost trajectories, corresponding to tail-end events of catastrophic failure, are more likely to be encountered by the GAIL-agents than the expert. This makes the reliability of GAIL-agents questionable when it comes to deployment in risk-sensitive applications like robotic surgery and autonomous driving. In this work, we aim to minimize the occurrence of tail-end events by minimizing tail risk within the GAIL framework. We quantify tail risk by the Conditional-Value-at-Risk (CVaR) of trajectories and develop the Risk-Averse Imitation Learning (RAIL) algorithm. We observe that the policies learned with RAIL show lower tail-end risk than those of vanilla GAIL. Thus, the proposed RAIL algorithm appears as a potent alternative to GAIL for improved reliability in risk-sensitive applications.
Subfields: Artificial Intelligence, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tom Everitt
- Sergey Levine
- Shane Legg
- Yilun Du
- Igor Mordatch
- Jan Leike
- Pieter Abbeel
- Antonio Torralba
In Reinforcement Learning
- Tom Everitt
- Jan Leike
- Shane Legg
- Sergey Levine
- Yilun Du
- Miljan Martic
- Igor Mordatch
- Antonio Torralba
Citations in the corpus
- Pre-Trained Language Models for Interactive Decision-Making (2022, methodology): “All our experiments use imitation learning (Santara et al., 2017; Ng et al., 2000; Peng et al., 2018), specifically”
- AI Safety Gridworlds (2017, background): “Another possible avenue could be the use of prior information, for example through imitation learning (Abbeel et al., 2010; Santara et al., 2017): the agent could try to “stay close” to the state space covered by demonstrations pro-”