Efficient Reductions for Imitation Learning

By Stéphane Ross, Drew Bagnell. 2010

1,044 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Efficiency & Compression

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…et al., 2019; Kumar et al., 2019a; Ghasemipour et al., 2021); imitation learning, where it is used to ﬁt an expert’s actions to obtain a policy (Ross & Bagnell, 2010; Ross et al., 2011); and other areas such as hierarchical RL (Peng et al., 2017; Co-Reyes et al., 2018; Jiang et al., 2019).”
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (2010, methodology): “1 as in Ross and Bagnell (2010), and for DA GGER the parameter β i = I ( i = 1) for I the indicator function.”
Generative Adversarial Imitation Learning (2016, background): “Behavioral cloning, while appealingly simple, only tends to succeed with large amounts of data, due to compounding error caused by covariate shift [23, 24].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “The proof follows from Theorem 2.1 from Ross et al. (2011) using the 0-1 loss, and the bound is the best possible bound following the example from Ross and Bagnell (2010).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Following work on Search-based Structured Prediction (SEARN) [137], Stochastic Mixing Iterative Learning (SMILE) trains a stochastic stationary policy over several iterations and then makes use of a geometric stochastic mixing of”
Deep Imitative Models for Flexible Inference, Planning, and Control (2018, methodology): “However, in practice, model-based and model-free RL algorithms are vulnerable to distributional drift (Thrun, 1995; Ross & Bagnell, 2010): when acting according to the learned model or policy, the agent visits states different from those seen during training, and in those it is unlikely to…”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, background): “The policy learned from behavioral cloning has compounding errors after we execute the policy for multiple steps and reach unseen states [Bagnell, 2015, Ross and Bagnell, 2010].”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, methodology): “This focus puts us closer to the supervised learning regime, and, indeed, a common approach to imitation learning – known as behavioral cloning (BC) – is to perform max-likelihood training on a collected a set of target demonstrations composed of state-action pairs sampled from the target policy [31”
Learning monocular reactive UAV control in cluttered natural environments (2012, background): “These data-driven approaches allow us to directly learn a control strategy that mimics an expert pilot’s choice of actions based on demonstrations of the desired behavior, i.e., sample flight trajectories through cluttered environments.”
An Algorithmic Perspective on Imitation Learning (2018, background): “Instead, a policy can be iteratively learned by alternating between policy updates and requesting additional demonstrations for the current state distribution [Ross and Bagnell, 2010, Ross et al., 2011, Bagnell, 2015].”
Improving Multi-Step Prediction of Learned Time Series Models (2015, methodology): “Furthermore, the bounds we have shown are valid with an infinite number of completely new trajectories at each step n, a similar requirement for related methods such as SEARN (Daumé III, Langford, and Marcu 2009), Forward training (Ross and Bagnell 2010), and DAGGER (Ross, Gordon, and Bagnell 2011).”
Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction (2017, background): “This is very difﬁcult to enforce in practice, and, as a result, policies learned by these methods can fail spectacularly in theory and in practice (Ross & Bagnell, 2010).”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, background): “A small mistake made by the agent due to this covariate shift (Ross and Bagnell, 2010; Ross et al., 2011) may lead to compounding errors which finally lead to failure of a task.”
Robust Imitation of Diverse Behaviors (2017, background): “These limitations of supervised learning for imitation, also known as behavioral cloning (BC) [24], are well known [27, 28].”
Reinforcement and Imitation Learning via Interactive No-Regret Learning (2014, background): “Iterative training procedures ( e.g. DA GGER , SEARN, SMIL E [24, 10, 21]) that interleave policy execution and learning have demonstrated impressive practical performance and strong theoretical guarantees that were not possible with batch supervised learning.”
Learning to Search Better than Your Teacher (2015, background): “A related discussion can be found at Theorem 2.1 in (Ross & Bagnell, 2010).”
Learning message-passing inference machines for structured prediction (2011, background): “In [17, 18], we have showed that this forward training procedure guarantees that the expected sum of the loss in the sequence is bounded byN̄, where ̄ is the average true loss of the learned predictors h1:N .”
DART: Noise Injection for Robust Imitation Learning (2017, methodology): “We measure the difference between controls using a surrogate loss l : U × U → R [14, 13].”