Query-Efficient Imitation Learning for End-to-End Autonomous Driving

By Jiakai Zhang, Kyunghyun Cho. 2016

238 citations recorded by the source corpus.

One way to approach end-to-end autonomous driving is to learn a policy function that maps from a sensory input, such as an image frame from a front-facing camera, to a driving action, by imitating an expert driver, or a reference policy. This can be done by supervised learning, where a policy function is tuned to minimize the difference between the predicted and ground-truth actions. A policy function trained in this way however is known to suffer from unexpected behaviours due to the mismatch between the states reachable by the reference policy and trained policy functions. More advanced algorithms for imitation learning, such as DAgger, addresses this issue by iteratively collecting training examples from both reference and trained policies. These algorithms often requires a large number of queries to a reference policy, which is undesirable as the reference policy is often expensive. In this paper, we propose an extension of the DAgger, called SafeDAgger, that is query-efficient and more suitable for end-to-end autonomous driving. We evaluate the proposed SafeDAgger in a car racing simulator and show that it indeed requires less queries to a reference policy. We observe a significant speed up in convergence, which we conjecture to be due to the effect of automated curriculum learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Learning Algorithms for Active Learning (2017, methodology): “These methods, which focus the cost of sampling from the oracle policy on states visited by the model policy, have recently been adopted by researchers working with deep networks for representation learning (Zhang & Cho, 2017; Sun et al., 2017).”
Agile Autonomous Driving using End-to-End Deep Imitation Learning (2017, methodology): “To our knowledge, DAgger has only been used heuristically in these domains [26, 33].”
DART: Noise Injection for Robust Imitation Learning (2017, background): “But despite their ability to address covariate shift, on-policy methods have been shown to be challenging for human supervisors [7] and require the robot to visit potentially dangerous regions of the state space during training [24].”
Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations (2016, background): “Zhang and Cho trained a classifier to detect when RC would encounter dangerous states during policy execution and have a human take over [22].”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, unknown): “Conventionally, RL algorithms designed to ensure safety [47, 26, 58, 69, 38, 17] model the environment as a Markov decision process (MDP), and rely upon the stationarity assumption made by MDPs [53].”
Active One-shot Learning (2017, background): “Applications have included domains where human or computer safety is at risk, such as cancer clas-siﬁcation [1], malware detection [2], and autonomous driving [3, 4].”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, unknown): “DAgger (Ross et al., 2011) achieves sublinear regret by repeatedly soliciting human action labels in an online fashion.”
DropoutDAgger: A Bayesian Approach to Safe Imitation Learning (2017, background): “In particular, SAFEDAGGER augments DAGGER with a decision rule policy to provide safe exploration with minimal influence from the expert [11].”
Backplay: "Man muss immer umkehren" (2018, unknown): “These methods explicitly encourage the learned policy to mimic an expert policy (Bain & Sommut, 1999; Ross et al., 2011; Daumé et al., 2009; Zhang & Cho, 2016; Laskey et al., 2016; Nair et al., 2017; Hester et al., 2017; Ho & Ermon, 2016; Aytar et al., 2018; Lerer & Peysakhovich, 2018; Peng et al.,…”
Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning (2020, methodology): “Our method for predicting failures is similar to but distinct from another popular method for reducing expert burden [14], in which the authors learn a “safety” policy that outputs whether an action is safe or not based on the 2 -norm between it and an expert action.”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, background): “SafeDAgger [49] reduced human involvement by a module deciding if the self-driving model needs expert’s help at any given moFig.”
Autoregressive Knowledge Distillation through Imitation Learning (2020, methodology): “We expect a similar strategy of mixing oracle and learner distributions to work well for non-convex neural networks, as shown in other applications (Zhang and Cho, 2016; Sun et al., 2017).”
InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations (2017, methodology): “[40] performs end-to-end visual imitation learning in TORCS through DAgger [18], querying the reference policies during training, which in many cases is difﬁcult.”
Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives (2023, methodology): “[49] propose the SafeDAgger algorithm, which intends to improve the query”
Planning and Decision-Making for Autonomous Vehicles (2018, background): “SafeDAgger (76) is a query-efficient extension to DAgger (77).”
HG-DAgger: Interactive Imitation Learning with Human Experts (2018, background): “Recent prior work using human experts has not fully addressed these issues, and has focused instead primarily on reducing the work load of the expert by minimizing the number of times the expert is queried [11]–[13].”