Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation

By Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, Yiannis Demiris. 2019

69 citations recorded by the source corpus.

We consider the problem of imitation learning from a finite set of expert trajectories, without access to reinforcement signals. The classical approach of extracting the expert's reward function via inverse reinforcement learning, followed by reinforcement learning is indirect and may be computationally expensive. Recent generative adversarial methods based on matching the policy distribution between the expert and the agent could be unstable during training. We propose a new framework for imitation learning by estimating the support of the expert policy to compute a fixed reward function, which allows us to re-frame imitation learning within the standard reinforcement learning setting. We demonstrate the efficacy of our reward function on both discrete and continuous domains, achieving comparable or better performance than the state of the art under different reinforcement learning algorithms.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

Hyperparameter Selection for Imitation Learning (2021, methodology): “As proposed by Burda et al. (2018); Wang et al. (2019), we define a support estimation metric by taking two randomly initialized networks, and train one to predict the random output of the other one on the expert training set.”
Energy-Based Imitation Learning (2020, methodology): “And we are unsuccessful with GMMIL and RED on Swimmer even after extensive tuning.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “The fundamental idea is to incentivize the agent to consistently remain within the support of the expert policy over time, prompting it to return to the demonstrated states when encountering new states outside the expert’s support [17], [26], [27].”
Continuous Control with Action Quantization from Demonstrations (2021, methodology): “Similarly a number of methods in Imitation Learning (Brantley et al., 2019; Wang et al., 2019) or in offline RL (Fujimoto & Gu, 2021; Wu et al., 2019) are evaluated on continuous control tasks and are based on Behavioral Cloning regularization which could be refined using the same type of…”
Primal Wasserstein Imitation Learning (2020, methodology): “…estimation Another line of research in IL consists in estimating the state-action support of the expert and deﬁne a reward that encourages the agent to stay on the support of the expert [Piot et al., 2014, Schroecker and Isbell, 2017, Wang et al., 2019, Brantley et al., 2020, Reddy et al., 2020].”