Imitation Learning via Off-Policy Distribution Matching

By Ilya Kostrikov, Ofir Nachum, Jonathan Tompson. 2019

250 citations recorded by the source corpus.

When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating distribution ratios and then using these ratios as rewards in a standard reinforcement learning (RL) algorithm. Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work to either be exorbitantly data- inefficient or alter the original objective in a manner that can drastically change its optimum. In this work, we show how the original distribution ratio estimation objective may be transformed in a principled manner to yield a completely off-policy objective. In addition to the data-efficiency that this provides, we are able to show that this objective also renders the use of a separate RL optimization unnecessary. Rather, an imitation policy may be learned directly from this objective without the use of explicit rewards. We call the resulting algorithm ValueDICE and evaluate it on a suite of popular imitation learning benchmarks, finding that it can achieve state-of-the-art sample efficiency and performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “…inference (Wainwright and Jordan, 2003), generative model estimation (Nowozin et al., 2016; Dai et al., 2019), imitation learning (Ke et al., 2019; Ghasemipour et al., 2019; Kostrikov et al., 2019), and reinforcement learning (Nachum et al., 2019a; Zhang et al., 2020; Nachum et al., 2019b).”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “(25) We note that using a single-sample estimate for the Bellman residual in general leads to biased gradients, although previous works have found this to not have a significant practical effect in these domains (Kostrikov et al., 2019).”
Implicit Behavioral Cloning (2021, methodology): “While algorithms like ValueDice [55] 238 implement distribution matching in a sample-efficient off-policy setting, they have not been proven on 239 image-observations or high degree-of-freedom action spaces.”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “In this work, we focus on imitation learning, where the aim is to learn how to act in the environment to match the behavior of some unknown target policy [25].”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, methodology): “E.g., until now, many open-sourced RL datasets [19, 22, 27] are shared as a set of state/action pairs comparable to example/label pairs in supervised learning.”
Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap (2021, methodology): “Existing work has explored various types of divergences including KL (Pomerleau 1989, Bojarski et al. 2016), Jensen-Shannon (Rhinehart et al. 2018), Reverse KL (Kostrikov et al. 2019), f (Ke et al. 2019), and Wasserstein (Dadashi et al. 2020).”