DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving

By Chenyi Chen, Ari Seff, Alain L. Kornhauser, Jianxiong Xiao. 2015

1,868 citations recorded by the source corpus.

Today, there are two major paradigms for vision-based autonomous driving systems: mediated perception approaches that parse an entire scene to make a driving decision, and behavior reflex approaches that directly map an input image to a driving action by a regressor. In this paper, we propose a third paradigm: a direct perception approach to estimate the affordance for driving. We propose to map an input image to a small number of key perception indicators that directly relate to the affordance of a road/traffic state for driving. Our representation provides a set of compact yet complete descriptions of the scene to enable a simple controller to drive autonomously. Falling in between the two extremes of mediated perception and behavior reflex, we argue that our direct perception representation provides the right level of abstraction. To demonstrate this, we train a deep Convolutional Neural Network using recording from 12 hours of human driving in a video game and show that our model can work well to drive a car in a very diverse set of virtual environments. We also train a model for car distance estimation on the KITTI dataset. Results show that our direct perception approach can generalize well to real driving images. Source code and data are available on our project website.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (2018, background): “For real-world technologies, such as self-driving cars [10], autonomous drones [14], and search-and-rescue robots [37], the test distribution may be non-stationary, and new observations will often be out-of-distribution (OoD), i.”
Real-World Image Datasets for Federated Learning (2019, background): “…2015 ] , object tracking [ Kang et al. , 2017 ] , etc. Object detection is able to provide valuable information for semantic understanding of images and videos, and is related to many applications, including face recognition, pedestrian detection and autonomous driving [ Chen et al. , 2015 ] .”
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (2017, background): “Therefore, deep learning systems have become prevalent in our lives, including security-sensitive applications such as face recognition [55], [62], fingerprint identification [66], spam filtering [64], [56], malware detection [21], [57], and autonomous vehicles [17].”
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (2017, background): “Convolutional neural networks (CNNs) in particular have been wildly successful for image processing tasks, and CNN-based image recognition models have been deployed to help identify plant and animal species [8] and autonomously drive cars [9].”
Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness Against Adversarial Attack (2018, background): “Deep Neural Networks (DNNs) have achieved great success in a variety of applications, including but not limited to image classification [23], speech recognition [17], machine translation [5], and autonomous driving [9].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “the moving commands, Giusti et al. (2016) and Tai et al. (2016) took the output of the Softmax layer multiplied with the moving commands coefﬁcients to smooth the output steering angles. DeepDriving (Chen et al., 2015) estimated several indicators for affordance of driving actions using CNN models, ”
Driving Policy Transfer via Modularity and Abstraction (2018, neutral): “pace between traditional modular pipelines and endto-end learning. Hadsell et al. [17] train a perception system via self-supervised learning and use it in a standard navigation pipeline. Chen et al. [6] decompose the driving problem into predicting affordances and then executing a policy based on t”
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation (2017, background): “dances [16], and end-to-end driving from demonstrations [17], [18].”
BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, methodology): “However, GCG only learned to avoid collisions, and CAPs and DeepDriving required human supervision in order to learn non-collision avoidance behaviors; in contrast, BADGR is able to reason about both geometric and non-geometric navigational without any human supervision in complex, real-world enviro”
A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, background): “[55]–[59] Direct supervised deep learning Imitates the target data: usually a human driver.”
A Fast Integrated Planning and Control Framework for Autonomous Driving via Imitation Learning (2017, background): “First, as pointed by [14], the decision-making level of the neural networks is too low to assure feasibility, safety and smoothness.”
CARLA: An Open Urban Driving Simulator (2017, background): “More recently, racing simulators have been used to evaluate new approaches to autonomous driving [28, 3].”
Playing for Data: Ground Truth from Computer Games (2016, methodology): “And an open-source driving simulator was used to learn mid-level cues for autonomous driving [10].”
Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “In context of navigation, learning and DRL has been used to obtain policies [4,12,13,21,35,50,56,59,64].”
IQA: Visual Question Answering in Interactive Environments (2017, background): “Visual Learning by Simulation: There has been an increased use of simulated environments and game platforms to train computer vision systems to perform tasks such as learning the dynamics of the world [47, 48, 70], semantic segmentation [18], pedestrian detection [43], pose estimation [53] and urban”
Textual Explanations for Self-Driving Vehicles (2018, background): “End-to-End Learning for Self-Driving Cars: Most of vehicle controllers for selfdriving cars can be divided in two types of approaches [5]: (1) a mediated perceptionbased approach and (2) an end-to-end learning approach.”
Visual Semantic Planning Using Deep Successor Representations (2017, methodology): “Computer games and simulated platforms have been used for training perceptual tasks, such as semantic segmentation [18], pedestrian detection [33], pose estimation [38], and urban driving [5, 41, 42, 45].”
Safe Visual Navigation via Deep Learning and Novelty Detection (2017, background): “While we focus on supervised learning of collision probability, there are many other ways to map between images and actions, including end-to-end neural networks [6, 19], affordance-based representations [7], reinforcement learning using either simulated or real images [23, 31], and classification o”
VirtualWorlds as Proxy for Multi-object Tracking Analysis (2016, background): “[27] is another example for basic building blocks of autonomous driving.”