End to End Learning for Self-Driving Cars
By Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, B. Flepp, Prasoon Goyal, L. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang and 2 more. 2016
4,658 citations recorded by the source corpus.
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands. This end-to-end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e.g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS).
Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Quoc V. Le
- Chelsea Finn
- Percy Liang
- Ruslan Salakhutdinov
In Automotive Engineering
- Sergey Levine
- Edouard Leurent
- Andreas Geiger
- Kashyap Chitta
- Raquel Urtasun
- Pieter Abbeel
- Lei Tai
- Pravendra Singh
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
- Andrea Vedaldi
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shixiang Shane Gu
- Shimon Whiteson
- Abhishek Gupta
- Yuxi Li
Citations in the corpus
- Recurrent Neural Networks for Semantic Instance Segmentation (2017, background): “Given enough training data and computational power, a great variety of automatic tasks such as object recognition [8], machine translation [9], speech recognition [10] or self-driving cars [11] have seen a boost of performance thanks to models trained end-to-end, i.”
- WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving (2019, background): “demonstrated end-to-end learning [2] for steering and recently it was applied to fisheye cameras [54].”
- On Calibration of Modern Neural Networks (2017, background): “As an example, consider a self-driving car that uses a neural network to detect pedestrians and other obstructions (Bojarski et al., 2016).”
- A General Framework for Uncertainty Estimation in Deep Learning (2019, neutral): “autonomous driving [29], [30] and flying [31].”
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Imitation learning has been a popular approach towards end-toend, data-driven methods in autonomous driving (Bojarski et al., 2016; Sun et al., 2018a; Pan et al., 2017; Codevilla et al., 2018).”
- Secure Evaluation of Quantized Neural Networks (2019, background): “nd in particular Convolutional Neural Networks (CNNs), have emerged as a promising solution for many real-life problems such as facial recognition [45], image and video analysis for self-driving cars [6] and even for playing games (most readers probably know of AlphaGo [58] which in 2016 beat one of”
- The Hidden Vulnerability of Distributed Learning in Byzantium (2018, background): “Beyond image recognition or video labeling for social networks, SGD–based machine learning is venturing into safety–critical applications, like health–care (Holzinger, 2016) and transportation (Bojarski et al., 2016).”
- Model-Reuse Attacks on Deep Learning Systems (2018, background): “We explore four deep learning systems used in securitycritical domains, including skin cancer screening [20], speech recognition [43], face verification [55], and autonomous steering [11].”
- Efficient Formal Safety Analysis of Neural Networks (2018, methodology): “Our model’s architecture is similar to the DAVE-2 self-driving car architecture from NVIDIA [6, 2] and it achieves similar 1-MSE as models used in [23].”
- AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation (2018, background): “Recent years have shown a wide adoption of deep neural networks in safety-critical applications, including self-driving cars [2], malware detection [44], and aircraft collision avoidance detection [21].”
- MagNet: A Two-Pronged Defense against Adversarial Examples (2017, background): “They are used in autonomous control for robots and vehicles [1, 3, 4], financial systems [32], medical treatments [31], information security [12, 29], and human-computer interaction [11, 13].”
- DeepXplore: Automated Whitebox Testing of Deep Learning Systems (2017, methodology): “Specifically, DAVE-orig [8] fully replicates the original architecture from the Nvidia’s paper [10].”
- Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks (2017, background): “Great effort is now being put into using DNNs as controllers for safety-critical systems such as autonomous vehicles [4] and airborne collision avoidance systems for unmanned aircraft (ACAS Xu) [13].”
- Safety Verification of Deep Neural Networks (2016, background): “Many applications are envisaged, including their use as perception modules and end-to-end controllers for self-driving cars [15].”
- Countering Adversarial Images using Input Transformations (2018, background): “inates 60% of strong gray-box and 90% of strong black-box attacks by a variety of major attack methods. 1 INTRODUCTION As the use of machine intelligence increases in security-sensitive applications (Bojarski et al., 2016; Amodei et al., 2015), robustness has become a critical feature to guarantee t”
- Self-Driving Car Steering Angle Prediction Based on Image Recognition (2019, methodology): “In order to establish a baseline for our research, we used the architecture from NVIDIA [1] to test different forms of data augmentation.”