ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst

By Mayank Bansal, Alex Krizhevsky, Abhijit S. Ogale. 2018

871 citations recorded by the source corpus.

Our goal is to train a policy for autonomous driving via imitation learning that is robust enough to drive a real vehicle. We find that standard behavior cloning is insufficient for handling complex driving scenarios, even when we leverage a perception system for preprocessing the input and a controller for executing the output on the car: 30 million examples are still not enough. We propose exposing the learner to synthesized data in the form of perturbations to the expert's driving, which creates interesting situations such as collisions and/or going off the road. Rather than purely imitating all data, we augment the imitation loss with additional losses that penalize undesirable events and encourage progress -- the perturbations then provide an important signal for these losses and lead to robustness of the learned model. We show that the ChauffeurNet model can handle complex situations in simulation, and present ablation experiments that emphasize the importance of each of our proposed changes and show that the model is responding to the appropriate causal factors. Finally, we demonstrate the model driving a car in the real world.

Subfields: Artificial Intelligence, Automotive Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

Citations in the corpus

Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “More recently, Chauf-feurnet [138] demonstrated the limits of imitation learning where even 30 million state-action samples were insufﬁcient to learn an optimal policy that mapped bird-eye view images (states) to control (action).”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “Deep neural network models have been used widely in a range of robotics applications [12, 13, 14, 15, 16, 17].”
Google Research Football: A Novel Reinforcement Learning Environment (2019, background): “s to train smart agents that can interact with their environment and solve complex tasks (Sutton and Barto, 2018). Real-world applications include robotics (Haarnoja et al., 2018), self-driving cars (Bansal, Krizhevsky, and Ogale, 2018), and control problems such as increasing the power efﬁciency of”
A Survey of Deep Learning Applications to Autonomous Vehicle Control (2019, background): “Moving away from end-to-end approaches, researchers at Waymo recently presented ChauffeurNet [142].”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “Bansal et al. [115] explore the underlying causes of particular outcomes by examining the ChauffeurNet model using manually crafted inputs that involve omitting particular objects.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “In end-to-end driving, Chauffeur-Net [244] addresses the causal confusion issue by using the past ego-motion as intermediate BEV abstractions and dropping out it with a 50% probability during training.”
StarNet: Targeted Computation for Object Detection in Point Clouds (2019, background): “Additionally, in SDC perception, every location in the scene is not equally important [50, 4, 2], and that importance can change dynamically based on the local environment and context.”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, background): “As we witness the blossom of end-to-end algorithms and success applications into various domains [38, 22], there are some attempt implementing such a philosophy in autonomous driving as well [41, 3, 15, 1, 12, 42, 53, 7, 8].”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, methodology): “We use a single time-step input since prior works on IL for autonomous driving have shown that using observation histories may not lead to performance gain [88], [89], [90], [91], [92].”
CoverNet: Multimodal Behavior Prediction Using Trajectory Sets (2019, methodology): “Numerous papers are evaluated purely on internal datasets [13, 35, 6, 28], as common public datasets are either relatively small [16], focused on highway driving [9], or are tangen-tially related to driving [32].”
HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction (2022, methodology): “Extensive works have used rasterized scenes as model inputs [4, 9, 12, 15, 19, 25, 43] and employed standard image models [11,24,26,44] for learning.”
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, neutral): “Such blind agents are undeniably dangerous, which highlights a broader concern: displacement metrics are not correlated to closed-loop driving [14, 17, 3, 16].”
Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning (2020, background): “IL learns a driving policy from expert driving data [3], [4], [6], [7].”
Learning by Cheating (2019, background): “In prior imitation learning approaches, data augmentation, namely multiple camera angles [5] or trajectory noise injection [2, 6, 11], was a crucial ingredient.”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, methodology): “ChauffeurNet’s [7] inputs are top-down HD maps containing both static information about the road and the locations of dynamic objects.”
Synthetic Data for Deep Learning (2019, background): “[35] discuss the insufficient data problem in imitation learning: for learning to drive, even 30 million real-world expert driving examples that combine into more than 60 days of driving is not sufficient to train an end-to-end driving model.”
Model-free Deep Reinforcement Learning for Urban Autonomous Driving (2019, background): “Bird-view representation for autonomous driving decision making is proposed recently to reduce the complexity of visual features [8], [2], [6].”