Query-Efficient Imitation Learning for End-to-End Simulated Driving

By Jiakai Zhang, Kyunghyun Cho. 2017

129 citations recorded by the source corpus.

One way to approach end-to-end autonomous driving is to learn a policy that maps from a sensory input, such as an image frame from a front-facing camera, to a driving action, by imitating an expert driver, or a reference policy. This can be done by supervised learning, where a policy is tuned to minimize the difference between the predicted and ground-truth actions. A policy trained in this way however is known to suffer from unexpected behaviours due to the mismatch between the states reachable by the reference policy and trained policy. More advanced algorithms for imitation learning, such as DAgger, addresses this issue by iteratively collecting training examples from both reference and trained policies. These algorithms often require a large number of queries to a reference policy, which is undesirable as the reference policy is often expensive. In this paper, we propose an extension of the DAgger, called SafeDAgger, that is query-efficient and more suitable for end-to-end autonomous driving. We evaluate the proposed SafeDAgger in a car racing simulator and show that it indeed requires less queries to a reference policy. We observe a significant speed up in convergence, which we conjecture to be due to the effect of automated curriculum learning.

Subfields: Artificial Intelligence, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “For imitation learning based systems, Safe DAgger [144]”
One-Shot Visual Imitation Learning via Meta-Learning (2017, background): “Imitation learning from raw pixels has been widely studied in the context of mobile robotics [12, 13, 14, 15].”
CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving (2018, background): “As a straight-forward idea, imitation learning that learns policies via supervised training on human driving data has been applied to a variety of tasks, including modeling navigational behavior [20], off-road driving [12,2], and road following [9,8,1,18,11].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “,severalpapers[10],[24],[25]have proposedaggregatingon-policydata[26]duringtraining.”
Exploring applications of deep reinforcement learning for real-world autonomous driving systems (2019, background): “SafeDAgger (Zhang and Cho, 2017) introduces a safety policy that learns to predict the error made by a primary policy trained initially with the supervised learning approach, without querying a reference policy.”
Exploring the Limitations of Behavior Cloning for Autonomous Driving (2019, background): “This problem is related to sensitivity to both initialization and sampling order [30], reproducibility issues in Reinforcement Learning [17, 28], and the need to move beyond the i.i.d. data assumption towards curriculum learning [5] for sensorimotor control [45, 3].”
Guided Meta-Policy Search (2019, unknown): “…algorithms and RL algorithms that leverage supervised learning have been far more successful (Levine et al., 2016; Bojarski et al., 2016; Giusti et al., 2016; Zhang & Cho, 2017), which can largely be attributed to the stability of supervised learning as compared with reinforcement learning.”
Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning (2020, methodology): “In [14], an oracle policy must still be available to be arbitrarily queried to update the safety policy, however, as is the case in DAgger.”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, background): “Zhang & Cho (2017) propose SafeDAgger a supervised learning approach to imitation learning that predicts in which states the imitation learning policy will have a large action difference from the demonstrator.”
Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data (2019, background): “At present, imitation learning has been applied to a variety of tasks, including autonomous flight [6], articulated motion [7], and end-to-end driving [8, 9].”
Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention (2018, background): “[60, 34, 32, 37] construct policies to minimize the number of queries to the teacher.”
Generative Multi-Agent Behavioral Cloning (2018, background): “We conclude that our model does not suffer from the Dying Units problem (Zhang et al., 2017; Chen et al., 2016), where the output policy model is sometimes sufficiently expressive on its own that the latent variables fail to encode anything meaningful.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “For example, SafeDAgger [23] uses an auxiliary safety policy to signal the agent to transfer the control to the expert when there is a likelihood of deviation from the expert’s trajectory.”