Learning a Driving Simulator

By Eder Santana, George Hotz. 2016

232 citations recorded by the source corpus.

this http URL's approach to Artificial Intelligence for self-driving cars is based on an agent that learns to clone driver behaviors and plans maneuvers by simulating future events in the road. This paper illustrates one of our research approaches for driving simulation. One where we learn to simulate. Here we investigate variational autoencoders with classical and learned cost functions using generative adversarial networks for embedding road frames. Afterwards, we learn a transition model in the embedded space using action conditioned Recurrent Neural Networks. We show that our approach can keep predicting realistic looking video for several frames despite the transition model being optimized without a cost function in the pixel space.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Self-Driving Car Steering Angle Prediction Based on Image Recognition (2019, background): “ai [14] has proposed to learn a driving simulator with an approach that combines a Variational Auto-encoder (VAE) [11] and a Generative Adversarial Network (GAN) [5].”
A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, neutral): “of their dataset [289] which includes 7.”
End-to-End Learning of Driving Models from Large-Scale Video Datasets (2016, background): “[21] propose to learn a driving simulator with an approach that combines a Variational Auto-encoder (VAE) [10] and a Generative Adversarial Network (GAN) [7].”
A Review on Deep Learning Techniques for Video Prediction (2020, background): “PredNet [75] 2017 [22], [188] stLSTMs [117], [119], [120], [137] RGB RGB X `1,`2 SR ? ? X X ContextVP [76] 2018 [88], [189] MD-LSTM [115], [117], [119], [120] RGB RGB X `1, GDL R ? ? 7 7”
A Commute in Data: The comma2k19 Dataset (2018, unknown): “There are several driving datasets in the literature such as KITTI [8], Cityscapes [9], RoboCar [10], ApolloScapes [11], Berkeley Deep Drive [12], including our previous public dataset [13].”
DDD17: End-To-End DAVIS Driving Dataset (2017, methodology): “They are inspired by LeCun’s early work (LeCun et al., 2005), the seminal open dataset from comma.ai (Santana & Hotz, 2016), and by recent Nvidia (Bojarski et al., 2016) and unpublished VW studies.”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, unknown): “ai 2016 [139] 1 X X X X X 7h 15min CC BY-NC-SA 3.”
Multi-task learning with deep model based reinforcement learning (2016, background): “The idea of learning predictive models has been previously proposed (Schmidhuber (2015); Santana & Hotz (2016)), but all of them focus on learning the predictive models in an unsupervised way.”
Unsupervised Real-to-Virtual Domain Unification for End-to-End Highway Driving (2018, methodology): “ai [25], which is estimated from real data.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, unknown): “ai 2016 [114] 2016 ✓ ✓ ✓ ✓ ✓ 7h 15min San Francisco CC BY-NC-SA 3.”
Exploring the Limitations of Behavior Cloning for Autonomous Driving (2019, background): “17,28], and the need to move beyond the i.i.d. data assumption towards curriculum learning [5] for sensorimotor control [45,3]. Driving benchmarks fall in two main categories: offline datasets, e.g., [13,35,43,16], or on-line environments. We focus here on on-line benchmarks, as visuomotor models pe”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “…heterogeneous information networks [184], privacy-preserving [185 - 188], social robot [189], cipher cracking [190], auxiliary automatic driving [236], continual learning [237], molecule development in oncology [238], GANs for finance [239-242], GANs for textile [243, 244], GANs for e-commerce…”
Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives (2023, background): “ai [143] enriches the diversity of data by additionally collecting localization information and control signals, so it can be implemented for more tasks, for example, localization and planning.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “MoCoGan [86] is proposed to decompose motion and content for video generation [348]–[350].”
Adversarially Robust Distillation (2019, background): “This phenomenon leads to major security vulnerabilities for highstakes applications like self-driving cars, medical diagnosis, and copyright control (Santana and Hotz 2016; Lee et al. 2017; Saadatpanah, Shafahi, and Goldstein 2019).”
Synthetic Data for Deep Learning (2019, background): “n segmentation masks [717], and Kuznichov et al. [340] generate synthetically augmented data that preserves the geometric structure of the leaves; all works report improved counting. Santana and Hotz [518] present a generative model that can learn to generate realistic looking images and even videos”
Multimodal End-to-End Autonomous Driving (2019, background): “steering angle, speed, geo-localization and orientation [85], [86], [65]); note that such supervision does not require human intervention in terms of explicitly annotating the content of the raw sensor data.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “Recent practices model the world state as raw sensor observations such as point clouds [7, 67, 140, 143, 147] and images [54, 61, 68, 90, 102, 125, 127, 131, 145].”
Generative adversarial networks: introduction and outlook (2017, background): “Santana et al. [41] propose to generate images with the same distribution as real driving scenarios.”