A Divergence Minimization Perspective on Imitation Learning Methods

By Seyed Kamyar Seyed Ghasemipour, Richard S. Zemel, Shixiang Shane Gu. 2019

292 citations recorded by the source corpus.

In many settings, it is desirable to learn decision-making and control policies through learning or bootstrapping from expert demonstrations. The most common approaches under this Imitation Learning (IL) framework are Behavioural Cloning (BC), and Inverse Reinforcement Learning (IRL). Recent methods for IRL have demonstrated the capacity to learn effective policies with access to a very limited set of demonstrations, a scenario in which BC methods often fail. Unfortunately, due to multiple factors of variation, directly comparing these methods does not provide adequate intuition for understanding this difference in performance. In this work, we present a unified probabilistic perspective on IL algorithms based on divergence minimization. We present $f$-MAX, an $f$-divergence generalization of AIRL [Fu et al., 2018], a state-of-the-art IRL method. $f$-MAX enables us to relate prior IRL methods such as GAIL [Ho & Ermon, 2016] and AIRL [Fu et al., 2018], and understand their algorithmic properties. Through the lens of divergence minimization we tease apart the differences between BC and successful IRL approaches, and empirically evaluate these nuances on simulated high-dimensional continuous control domains. Our findings conclusively identify that IRL's state-marginal matching objective contributes most to its superior performance. Lastly, we apply our new understanding of IL methods to the problem of state-marginal matching, where we demonstrate that in simulated arm pushing environments we can teach agents a diverse range of behaviours using simply hand-specified state distributions and no reward functions or expert demonstrations. For datasets and reproducing results please refer to this https URL .

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Additionally, a unified probabilistic perspective on f -divergence minimization in imitation learning has been presented for both discrete and continuous control environments (Ke et al., 2021; Ghasemipour et al., 2020).”
Imitating Interactive Intelligence (2020, neutral): “Instead of matching πθ(s, a) = π∗(s, a), one can instead attempt to enforce only that πθ(s) = π∗(s) (Merel et al., 2017; Ghasemipour et al., 2020).”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “…inference (Wainwright and Jordan, 2003), generative model estimation (Nowozin et al., 2016; Dai et al., 2019), imitation learning (Ke et al., 2019; Ghasemipour et al., 2019; Kostrikov et al., 2019), and reinforcement learning (Nachum et al., 2019a; Zhang et al., 2020; Nachum et al., 2019b).”
Hyperparameter Selection for Imitation Learning (2021, methodology): “Although previous work has identified measures of similarity not based on the reward functions (Dadashi et al., 2020; Ghasemipour et al., 2020; Ke et al., 2019), this work is, as far as we know, the first to propose a principled evaluation protocol to select HPs not based on the true but supposedly…”
Energy-Based Imitation Learning (2020, methodology): “4Full derivations can be found in Appendix C and D of [14] and without loss of generality we replace c (s, a) with −r (s, a) .”
What Matters for Adversarial Imitation Learning? (2021, methodology): “Following prior work [14, 31, 46], we subsample expert demonstrations by only using every 20th state-action pair to make the tasks harder.”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “Indeed, we use diffusion models as a practical BC approach that optimise a variational bound of the usual BC objective (which can be interpreted as minimising the forward KL w.r.t. to the experts distribution Ghasemipour et al. (2020); Ke et al. (2020)), but, as discussed in Sec.”
Action and Perception as Divergence Minimization (2020, background): “Ghasemipour et al. (2019) describe imitation learning as minimizing divergences between the inputs of learned and expert behavior.”
Primal Wasserstein Imitation Learning (2020, methodology): “These methods are well suited to the IL problem since they implicitely minimize an f -divergence between the state-action distribution of an expert and the state-action distribution of the learning agent [Ghasemipour et al., 2019, Ke et al., 2019].”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, background): “…approaches to imitation learning are typically based on maximum entropy policy optimization and divergence minimization between marginal state-action distributions (Ho & Ermon, 2016; Fu et al., 2017; Ghasemipour et al., 2019) and are related to Generative Adversarial Networks (Finn et al., 2016a).”
Continuous Control with Action Quantization from Demonstrations (2021, background): “A number of existing works (Ho & Ermon, 2016; Ghasemipour et al., 2019; Dadashi et al., 2021) cast the problem into matching the state distributions of the agent and of the expert.”