Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction

By Wen Sun, Arun Venkatraman, Geoffrey J. Gordon, Byron Boots, J. Bagnell. 2017

247 citations recorded by the source corpus.

Recently, researchers have demonstrated state-of-the-art performance on sequential prediction problems using deep neural networks and Reinforcement Learning (RL). For some of these problems, oracles that can demonstrate good performance may be available during training, but are not used by plain RL methods. To take advantage of this extra information, we propose AggreVaTeD, an extension of the Imitation Learning (IL) approach of Ross & Bagnell (2014). AggreVaTeD allows us to use expressive differentiable policy representations such as deep networks, while leveraging training-time oracles to achieve faster and more accurate solutions with less training data. Specifically, we present two gradient procedures that can learn neural network policies for several problems, including a sequential prediction task and several high-dimensional robotics control problems. We also provide a comprehensive theoretical study of IL that demonstrates that we can expect up to exponentially-lower sample complexity for learning with AggreVaTeD than with plain RL algorithms. Our results and theory indicate that IL (and AggreVaTeD in particular) can be a more effective strategy for sequential prediction than plain RL.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “, 2017), and modified policy gradient approaches (Ho et al., 2016; Sun et al., 2017; Cheng et al., 2018; Sun et al., 2018).”
Learning Algorithms for Active Learning (2017, methodology): “These methods, which focus the cost of sampling from the oracle policy on states visited by the model policy, have recently been adopted by researchers working with deep networks for representation learning (Zhang & Cho, 2017; Sun et al., 2017).”
Deep Q-learning From Demonstrations (2017, background): “Deeply AggreVaTeD (Sun et al. 2017) extends DAGGER to work with deep neural networks and continuous action spaces.”
Dual Policy Iteration (2018, methodology): “(c) when an optimal expert policy ⇡⇤ is available during and only during training, we can set every ⌘n to be ⇡⇤, and DPI then reveals a previous IL algorithm—AGGREVATED [27].”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “, 2011], AggreVaTe [Ross and Bagnell, 2014], AggreVaTeD [Sun et al., 2017], DART [Laskey et al.”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, background): “However, it is not possible to compute this quantity without additional rollouts or assumptions [46].”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “Task-level abstration Takano and Nakamura [2015], Niekum et al.”
Optimal Completion Distillation for Sequence Learning (2018, methodology): “AggreVaTeD (Sun et al., 2017) assumes access to an unbiased estimate of Q-values and relies on techniques such as conjugate gradient and variance reduction regularization.”
DART: Noise Injection for Robust Imitation Learning (2017, neutral): “Recently, Sun et al. proposed a gradient update for on-policy methods to alleviate the computation burden [20].”
Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning (2018, methodology): “The fourth expression in the above equation is exactly the gradient proposed by AGGREVATED (Sun et al., 2017).”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “In general, imitation learning (Daum´e III et al., 2009; Ross et al., 2011; Ross and Bagnell, 2014; Chang et al., 2015; Sun et al., 2017) ﬁnds a policy ˆ π that minimizes the expected imitation loss L with respect to a teacher policy π (cid:63) under the agent-induced state distribution D ˆ π : We…”
Hierarchical Imitation and Reinforcement Learning (2018, background): “The latter setting (Daumé et al., 2009; Ross et al., 2011; Ross & Bagnell, 2014; Chang et al., 2015; Sun et al., 2017) assumes an interactive expert that provides demonstrations in response to actions taken by the current policy.”
Autoregressive Knowledge Distillation through Imitation Learning (2020, methodology): “We expect a similar strategy of mixing oracle and learner distributions to work well for non-convex neural networks, as shown in other applications (Zhang and Cho, 2016; Sun et al., 2017).”
Ranking Policy Gradient (2019, methodology): “Many eﬀorts have focused on developing the connections between RL and supervised learning, such as Expectation-Maximization algorithms (Dayan and Hinton, 1997; Peters and Schaal, 2007; Kober and Peters, 2009; Abdolmaleki et al., 2018), Entropy-Regularized RL (Oh et al., 2018; Haarnoja et al., 2018),”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “[93] Sim State, Image Interactive IL Online Yes Robotic Locomotion, Dependency Parsing”
On the Sample Complexity of Stability Constrained Imitation Learning (2021, background): “IL oﬀers many appealing advantages: it is often more sample-eﬃcient than reinforcement learning [3, 4], and can lead to policies that are more computationally eﬃcient to evaluate online [5, 6] than optimization-based experts.”
Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation (2019, background): “While several strategies have been proposed to address this Ross & Bagnell (2010); Sun et al. (2017), they often require access to the expert policy during the entire training process, rather than a ﬁnite set of expert trajectories.”
Predictive-State Decoders: Encoding the Future into Recurrent Networks (2017, methodology): “We applied the method of AggreVaTeD [46], a policy-gradient method, to train our expert models.”