Imitation Learning as f-Divergence Minimization

By Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, Siddhartha S. Srinivasa. 2019

193 citations recorded by the source corpus.

We address the problem of imitation learning with multi-modal demonstrations. Instead of attempting to learn all modes, we argue that in many tasks it is sufficient to imitate any one of them. We show that the state-of-the-art methods such as GAIL and behavior cloning, due to their choice of loss function, often incorrectly interpolate between such modes. Our key insight is to minimize the right divergence between the learner and the expert state-action distributions, namely the reverse KL divergence or I-projection. We propose a general imitation learning framework for estimating and minimizing any f-Divergence. By plugging in different divergences, we are able to recover existing algorithms such as Behavior Cloning (Kullback-Leibler), GAIL (Jensen Shannon) and Dagger (Total Variation). Empirical results show that our approximate I-projection technique is able to imitate multi-modal behaviors more reliably than GAIL and behavior cloning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Additionally, a unified probabilistic perspective on f -divergence minimization in imitation learning has been presented for both discrete and continuous control environments (Ke et al., 2021; Ghasemipour et al., 2020).”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “…variational inference (Wainwright and Jordan, 2003), generative model estimation (Nowozin et al., 2016; Dai et al., 2019), imitation learning (Ke et al., 2019; Ghasemipour et al., 2019; Kostrikov et al., 2019), and reinforcement learning (Nachum et al., 2019a; Zhang et al., 2020; Nachum et…”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “This matching is done by minimizing their Jensen-Shannon divergence between the two occupancy measures (Ho and Ermon, 2016; Ke et al., 2020).”
A Divergence Minimization Perspective on Imitation Learning Methods (2019, background): “Concurrent to our work, Ke et al. [34] also present a unifying probabilistic perspective on IL; however, their empirical experiments solely focus on grid world domains, while our work provides comparative results on high-dimensional continuous control environments and also evaluates the…”
Imitation Learning via Off-Policy Distribution Matching (2019, methodology): “Distribution matching via adversarial learning, or Adversarial Imitation Learning (AIL), has recently become a popular approach for imitation learning (Ho & Ermon, 2016; Fu et al., 2017; Ke et al., 2019; Kostrikov et al., 2019).”
Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap (2021, background): “Additionally, in sharp contrast to recently proposed approaches (Ke et al. 2019, Kostrikov et al. 2019, Jarrett et al. 2020, Rhinehart et al. 2018), moments, due to their simple forms as expectations of basis functions, can be effectively estimated via demonstrator samples and the uncertainty in…”
Primal Wasserstein Imitation Learning (2020, methodology): “These methods are well suited to the IL problem since they implicitely minimize an f -divergence between the state-action distribution of an expert and the state-action distribution of the learning agent [Ghasemipour et al., 2019, Ke et al., 2019].”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “These both suffer due to the approximations they make (illustrated in Figure 1), either encouraging the agent to learn an ‘average’ policy or predicting action dimensions independently resulting in ‘uncoordinated’ behaviour (Ke et al., 2020).”