End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners

By Simon Hecker, Dengxin Dai, Luc Van Gool. 2018

179 citations recorded by the source corpus.

For human drivers, having rear and side-view mirrors is vital for safe driving. They deliver a more complete view of what is happening around the car. Human drivers also heavily exploit their mental map for navigation. Nonetheless, several methods have been published that learn driving models with only a front-facing camera and without a route planner. This lack of information renders the self-driving task quite intractable. We investigate the problem in a more realistic setting, which consists of a surround-view camera system with eight cameras, a route planner, and a CAN bus reader. In particular, we develop a sensor setup that provides data for a 360-degree view of the area surrounding the vehicle, the driving route to the destination, and low-level driving maneuvers (e.g. steering angle and speed) by human drivers. With such a sensor setup we collect a new driving dataset, covering diverse driving scenarios and varying weather/illumination conditions. Finally, we learn a novel driving model by integrating information from the surround-view cameras and the route planner. Two route planners are exploited: (1) by representing the planned routes on OpenStreetMap as a stack of GPS coordinates, and (2) by rendering the planned routes on TomTom Go Mobile and recording the progression into a video. Our experiments show that: (1) 360-degree surround-view cameras help avoid failures made with a single front-view camera, in particular for city driving and intersection scenarios; and (2) route planners help the driving task significantly, especially for steering angle prediction. Code, data and more visual results will be made available at http://www.vision.ee.ethz.ch/~heckers/Drive360.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, background): “However, robust automated driving in urban environments has not been achieved yet [14].”
A survey of deep learning techniques for autonomous driving (2019, methodology): “In Hecker et al. (2018), surround‐view cameras were used for End2End learning.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, unknown): “Drive360 [113] 2019 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 55h Switzerland Academic”
Predicting Semantic Map Representations From Images Using Pyramid Occupancy Networks (2020, background): “[12] augment their camera-based end-to-end driving model with a rendered map view from a commercial GPS route planner and show that this significantly improves driving performance.”
When2com: Multi-Agent Perception via Communication Graph Grouping (2020, methodology): “Since current semantic segmentation datasets [9, 5, 26, 14] only provide RGB images and labels captured from the perspective of single agent, we thus use AirSim simulator [32] to collect our AirSim-MAP (Multi-Agent Perception) dataset.”
Variational End-to-End Navigation and Localization (2018, methodology): “Some recent methods incorporate maps as inputs [5], [6] to capture longer term action structure, yet they ignore the uncertainty maps inherently allow us to address – uncertainty about the location, and uncertainty about the longer-term plan.”
Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020, methodology): “While selecting features known to be associated with disease is meaningful, future work may further bene ﬁ t from utilizing large volumes of feature-rich data, as seen in ﬁ elds outside medicine such as autonomous driving 44,45 .”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, methodology): “coordinates as input was not as effective as using a route planner screen [8].”
A Survey of Deep Learning Applications to Autonomous Vehicle Control (2019, methodology): “[137] Supervised Learning CNN 360-degree view camera image, route plan Steering angle, velocity Takes route plan into account Lack of live testing No live testing, tested on data set image examples only”
Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding (2018, background): “Adverse weather conditions create visibility problems for both people and the sensors that power automated systems [25,37,48].”
Argoverse: 3D Tracking and Forecasting With Rich Maps (2019, methodology): “[13] show that end-toend route planning can be improved by processing rasterized maps from OpenStreetMap and TomTom.”
Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime (2018, background): “Recent years have witnessed great progress in autonomous driving [11], resulting in announcements that autonomous vehicles have driven over many thousands of miles and that companies aspire to sell such vehicles in a few years.”