End-to-End Learning of Driving Models from Large-Scale Video Datasets

By Huazhe Xu, Yang Gao, Fisher Yu, Trevor Darrell. 2016

877 citations recorded by the source corpus.

Robust perception-action models should be learned from training data with diverse visual appearances and realistic behaviors, yet current approaches to deep visuomotor policy learning have been generally limited to in-situ models learned from a single vehicle or simulation environment. We advocate learning a generic vehicle motion model from large scale crowd-sourced video data, and develop an end-to-end trainable architecture for learning to predict a distribution over future vehicle egomotion from instantaneous monocular camera observations and previous vehicle state. Our model incorporates a novel FCN-LSTM architecture, which can be learned from large-scale crowd-sourced vehicle action data, and leverages available scene segmentation side tasks to improve performance under a privileged learning paradigm. We provide a novel large-scale dataset of crowd-sourced driving behavior suitable for training our model, and report results predicting the driver action on held out sequences across diverse conditions.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

In Robotics & Control

Citations in the corpus

Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Discretisation in log-space has also been suggested, as many steering angles which are selected in practice are close to the centre [77].”
Driving Policy Transfer via Modularity and Abstraction (2018, methodology): “For training, we use standard public datasets – CamVid [5], Cityscapes [9], Berkeley Driving [52], and a combination of all these – as well as our small dataset and data collected in simulation.”
A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, methodology): “A spatiotemporal network, an FCN-LSTM architecture, was developed for predicting ego-vehicle motion in [63].”
Reinforcement Learning from Imperfect Demonstrations (2018, background): “Imitation learning is one of such approaches, examples including (Xu et al., 2016; Bojarski et al., 2016).”
Textual Explanations for Self-Driving Vehicles (2018, methodology): “As for the end-to-end approaches, recent works [3,26] suggest that neural networks can be successfully applied to self-driving cars in an end-to-end manner.”
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning (2018, methodology): “Our data was collected from the same original source as the videos in [33].”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, background): “Other works include learning spatial-temporal information for precipitation prediction [4], physical interaction [5], and autonomous driving [6].”
Gibson Env: Real-World Perception for Embodied Agents (2018, background): “Similarly, video datasets [53, 66, 95] are pre-recorded and still passive.”
Learned Step Size Quantization (2019, background): “networks are emerging as components of a number of revolutionary technologies, including image recognition (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012), and driving assistance (Xu et al., 2017). Unlocking the full promise of such applications requires a system perspective wher”
Exploring the Limitations of Behavior Cloning for Autonomous Driving (2019, neutral): “[43] H.”
MODNet: Motion and Appearance based Moving Object Detection Network for Autonomous Driving (2018, background): “(2) The behavior reflex approach that learns end to end the driving decision [2] [33].”
End to End Vehicle Lateral Control Using a Single Fisheye Camera (2018, background): “Since then, a lot of work has been conducted offline [6] [7] [8] on the Udacity challenge and dataset [9].”
GFF: Gated Fully Fusion for Semantic Segmentation (2019, background): “…category for every pixel in an image, such comprehensive image understanding is valuable for many vision-based applications such as medical image analysis (Ronneberger, Fischer, and Brox 2015), remote sensing (Kampffmeyer, Sal-berg, and Jenssen 2016) and autonomous driving (Xu et al. 2017).”
Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention (2017, background): “[28] proposed an end-to-end egomotion prediction approach that takes raw pixels and prior vehicle state signals as inputs and predicts a sequence of discretized actions (i.”
LiDAR-Video Driving Dataset: Learning Driving Policies Effectively (2018, methodology): “Recently, [27] broadened video scenes and illustrated that it is feasible for vehicles to drive in multiple complex situations.”
Learning Navigation Subroutines from Egocentric Videos (2019, neutral): “One way to do this is by instrumenting the agent to collect the perceptual input as well as the action executed, as done in autonomous driving [3].”
Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues (2017, methodology): “The most relevant to ours is the work in [8].”
Virtual to Real Reinforcement Learning for Autonomous Driving (2017, background): “However, supervised approach usually requires large amount of data to train a model [31] that can generalize to different environments.”