Exploring the Limitations of Behavior Cloning for Autonomous Driving

By Felipe Codevilla, Eder Santana, Antonio M. López, Adrien Gaidon. 2019

703 citations recorded by the source corpus.

Driving requires reacting to a wide variety of complex environment conditions and agent behaviors. Explicitly modeling each possible scenario is unrealistic. In contrast, imitation learning can, in theory, leverage data from large fleets of human-driven cars. Behavior cloning in particular has been successfully used to learn simple visuomotor policies end-to-end, but scaling to the full spectrum of driving behaviors remains an unsolved problem. In this paper, we propose a new benchmark to experimentally investigate the scalability and limitations of behavior cloning. We show that behavior cloning leads to state-of-the-art results, executing complex lateral and longitudinal maneuvers, even in unseen environments, without being explicitly programmed to do so. However, we confirm some limitations of the behavior cloning approach: some well-known limitations (e.g., dataset bias and overfitting), new generalization issues (e.g., dynamic objects and the lack of a causal modeling), and training instabilities, all requiring further research before behavior cloning can graduate to real-world driving. The code, dataset, benchmark, and agent studied in this paper can be found at \url{github.com/felipecode/coiltraine/blob/master/docs/exploring_limitations.md}

Subfields: Artificial Intelligence, Automotive Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

Citations in the corpus

Parting with Misconceptions about Learning-based Vehicle Motion Planning (2023, methodology): “We refer to this setting as ego-forecasting [2, 3, 13, 14].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Notable progress has been achieved with the development of deep neural networks, both in imitation learning [15], [16] and reinforcement learning [4], [17], [18], [19].”
MP3: A Unified Model to Map, Perceive, Predict and Plan (2021, background): “Although simple and general, such methods of directly generating control command from sensor data may have stability and robustness issues [12].”
Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer (2022, background): “A series of work follow this way, yielding conditional end-to-end driving [14, 15] in urban scenarios.”
Overcoming Model Bias for Robust Offline Deep Reinforcement Learning (2020, methodology): “These methods face difﬁculties once the trained policy leaves the known region of the state-action space on the real system—a problem referred to as covariate shift (Ho and Ermon 2016; Codevilla et al. 2019).”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “…that generates the control signal directly from perception input, whereas a modular pipeline consists of various sub-modules, each with task-speciﬁc functionalities. and Ranging (LiDAR) [6], [7], [10], navigation commands [23], [34], [36], and vehicle dynamics, such as speed [35], [37], [38].”
Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits (2023, background): “Compared with AEM + AIL, AEM + VM has superior performance in most cases; one interpretation could be that the value modeling provides better generalization ability, while simply imitating the aligned data can lead to accumulated off-track errors in unseen contexts (Codevilla et al., 2019).”
Tool Learning with Foundation Models (2023, neutral): “Besides, it is impractical to explicitly annotate every possible scenario of environment condition and agent behavior (Codevilla et al., 2019).”
Behavior Transformers: Cloning k modes with one stone (2022, positive): “When demonstration data is plentiful, such approaches have found impressive success in a variety of domains from self-driving [68, 14] to robotic manipulation [84, 62].”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, background): “As Figure 2 shows, this causes difficulties generating off-center deviations, known as the “lack-of-exploration” problem in behavior cloning [8].”
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, methodology): “CILRS [18] is proposed afterwards and further introduces a speed prediction head.”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, background): “reluctant to change its speed, for example to start moving again after stopping behind another car or a at traffic light [42].”
Learning by Cheating (2019, methodology): “The NoCrash benchmark [7] consists of three driving conditions, each on a shared set of 25 predefined routes with comparable difficulty to the full navigation condition in the CoRL2017 benchmark.”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, methodology): “One direction of end-to-end pipelines is to focus on the ultimate planning task mainly without explicit design of the intermediate representation [13, 14, 42, 9, 12, 26].”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, methodology): “CILRS [38] is a widely used image-only baseline for the old CARLA version 0.8.4.”
VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, background): “Early attempts [39, 8, 41] use a completely black-box spirit, where sensor data is directly used to predict control signals.”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “However, the direct generation of control outputs from sensor data poses significant challenges in robustness and safety assurance [11].”
Planning-oriented Autonomous Driving (2022, background): “However, for such methods of directly generating control outputs from sensor data, the transfer from the synthetic environment to realistic application remains a problem considering their robustness and safety assurance [22, 38].”