Reinforcement and Imitation Learning via Interactive No-Regret Learning

By Stéphane Ross, J. Bagnell. 2014

290 citations recorded by the source corpus.

Recent work has demonstrated that problems-- particularly imitation learning and structured prediction-- where a learner's predictions influence the input-distribution it is tested on can be naturally addressed by an interactive approach and analyzed using no-regret online learning. These approaches to imitation learning, however, neither require nor benefit from information about the cost of actions. We extend existing results in two directions: first, we develop an interactive imitation learning approach that leverages cost information; second, we extend the technique to address reinforcement learning. The results provide theoretical support to the commonly observed successes of online approximate policy iteration. Our approach suggests a broad new family of algorithms and provides a unifying view of existing techniques for imitation and reinforcement learning.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Thinking Fast and Slow with Deep Learning and Tree Search (2017, methodology): “We can also recover a version of Policy Iteration [14] by using Monte Carlo Search as our expert; in this case it is easy to see that Monte Carlo Tree Search gives stronger plan than does Monte Carlo Search.”
Learning Algorithms for Active Learning (2017, methodology): “Related approaches have been explored in previous work on imitation learning and learning to search (Ross & Bagnell, 2014; Chang et al., 2015).”
Deep Imitative Models for Flexible Inference, Planning, and Control (2018, background): “While our target setting is ofﬂine data collection, online imitation learning is an active area of research in the case of hybrid IL-RL (Ross & Bagnell, 2014; Sun et al., 2018) and “safe” IL (Sun et al., 2017; Menda et al., 2017; Zhang & Cho, 2017).”
Dual Policy Iteration (2018, methodology): “, we are performing imitation learning by treating ⌘n as an expert in this iteration [26, 27].”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “…or an approximate value function, in the training time or in the phase of collecting demonstrations, then, stronger algorithms can be designed, such as DAgger [Ross et al., 2011], AggreVaTe [Ross and Bagnell, 2014], AggreVaTeD [Sun et al., 2017], DART [Laskey et al., 2017], THOR Sun et al. [2018a].”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “Furthermore, off-policy deep RL can be unstable due to the compounding effects of bootstrapping value learning and function approximation [2, 5, 11, 33].”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “Modern, high performance, implementations of such search based structured prediction use online learning methods of DAGGER [Ross et al., 2011, 2013], AggreVaTe [Ross and Bagnell, 2014, Sun et al., 2017], or LOLS [Chang et al., 2015, Daumé III and Langford, 2015].”
Optimal Completion Distillation for Sequence Learning (2018, methodology): “Our work is inspired by Imitation Learning techniques (Ross et al., 2011; Ross and Bagnell, 2014; Sun et al., 2018), where a student policy is optimized to imitate an expert teacher.”
Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction (2017, methodology): “When we use non-linear function approximators to represent the polices, the analysis of AggreVaTe from (Ross & Bagnell, 2014) will not hold, since the loss function (cid:96) n ( θ ) is not convex with respect to parameters θ .”
Learning to Search Better than Your Teacher (2015, positive): “…is learning to search ( L 2 S ) (Collins & Roark, 2004; Daum´e III & Marcu, 2005; Daum´e III et al., 2009; Ross et al., 2011; Doppa et al., 2014; Ross & Bagnell, 2014), which solves the problem by: 1. converting structured prediction into a search problem with speciﬁed search space and…”
Batch Policy Learning under Constraints (2019, background): “This approach provides a bridge between RL and IL, whose objective functions are fundamentally different (see AggreVate by (Ross & Bagnell, 2014) for an alternative approach).”
Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning (2018, methodology): “In IL, the learner can reference expert demonstrations (Abbeel & Ng, 2004), or can access a cost-to-go oracle (Ross & Bagnell, 2014), providing additional information about the long-term effects of learner decisions.”
Query-Efficient Imitation Learning for End-to-End Simulated Driving (2017, background): “In AggreVaTe (Ross and Bagnell 2014), for instance, the roll-out by a reference policy may be executed not from a uniformrandomly selected time point, but from the time step when the safety classifier returns 0.”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “imitation learning algorithms (Daumé III et al., 2009; Ross et al., 2011; Ross and Bagnell, 2014; Chang et al., 2015; Sun et al., 2017; Sharaf and Daumé III, 2017) does not take into consideration the agent’s capability and behavior.”
Query-Efficient Imitation Learning for End-to-End Autonomous Driving (2016, background): “However, it is certainly possible to incorporate an explicit reward with other imitation learning algorithms, such as SEARN [7], AggreVaTe [15] and LOLS [5].”
Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations (2016, background): “One approach is to consider the value of each action and select actions that lead to higher reward during roll-out [18].”
Imitation Learning as f-Divergence Minimization (2019, methodology): “Even though IL has of late been used to bootstrap reinforcement learning (RL) [11–15], we focus on the original problem where an extrinsic reward is not deﬁned.”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “Unfortunately, 66 DAgger and some of its variants [27, 28] are notoriously difficult to apply to robotic manipulation 67 because they necessitate an interface where the expert must annotate the correct action when not in 68 control of the robot policy.”
On Oracle-Efficient PAC RL with Rich Observations (2018, methodology): “The oracle-based approach has produced practically effective algorithms for active learning [18], contextual bandits [19], structured prediction [20, 21], and multi-class classification [22], and here, we consider oracle-based algorithms for challenging RL settings.”