A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

By Stéphane Ross, Geoffrey J. Gordon, J. Bagnell. 2010

4,272 citations recorded by the source corpus.

Sequential prediction problems such as imitation learning, where future observations depend on previous predictions (actions), violate the common i.i.d. assumptions made in statistical learning. This leads to poor performance in theory and often in practice. Some recent approaches provide stronger guarantees in this setting, but remain somewhat unsatisfactory as they train either non-stationary or stochastic policies and require a large number of iterations. In this paper, we propose a new iterative algorithm, which trains a stationary deterministic policy, that can be seen as a no regret algorithm in an online learning setting. We show that any such no regret algorithm, combined with additional reduction assumptions, must find a policy with good performance under the distribution of observations it induces in such sequential settings. We demonstrate that this new approach outperforms previous approaches on two challenging imitation learning problems and a benchmark sequence labeling problem.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Mohit Bansal
  4. Pieter Abbeel
  5. Jianfeng Gao
  6. Ruslan Salakhutdinov
  7. Percy Liang
  8. Quoc V. Le

In Management Science and Operations Research

  1. Sergey Levine
  2. Pieter Abbeel
  3. Ofir Nachum
  4. Chelsea Finn
  5. Shimon Whiteson
  6. Nan Jiang
  7. R´emi Munos
  8. Lihong Li

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Ofir Nachum
  5. Shimon Whiteson
  6. Shixiang Shane Gu
  7. Yuxi Li
  8. Abhishek Gupta

In Optimization & Training

  1. Yoshua Bengio
  2. Sergey Levine
  3. Pieter Abbeel
  4. Quoc V. Le
  5. Chelsea Finn
  6. Ruslan Salakhutdinov
  7. Razvan Pascanu
  8. Kyunghyun Cho

Citations in the corpus