Learning to Drive in a Day

By Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John M. Allen, Vinh-Dieu Lam, Alex Bewley, Amar Shah. 2018

800 citations recorded by the source corpus.

We demonstrate the first application of deep reinforcement learning to autonomous driving. From randomly initialised parameters, our model is able to learn a policy for lane following in a handful of training episodes using a single monocular image as input. We provide a general and easy to obtain reward: the distance travelled by the vehicle without the safety driver taking control. We use a continuous, model-free deep reinforcement learning algorithm, with all exploration and optimisation performed on-vehicle. This demonstrates a new framework for autonomous driving which moves away from reliance on defined logical rules, mapping, and direct supervision. We discuss the challenges and opportunities to scale this approach to a broader range of autonomous driving tasks.

Subfields: Artificial Intelligence, Automotive Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Reinforcement learning approaches have been applied in both simulation (Sallab et al., 2017) and in the real world, with human interventions in case the vehicle violates a safety constraint (Kendall et al., 2019) (see Figure 7).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, neutral): “[92].”
BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, background): “…learning navigation policies directly from real-world experience, but typically require a person [18], [21] or SLAM algorithm [22] to gather the data, assume access to the ground-truth robot state [23], learn using low-bandwidth sensors [24], or only perform collision avoidance [4], [25].”
A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, background): “There are three main approaches for end-to-end driving: direct supervised deep learning [55]–[59], neuroevolution [62], [63] and the more recent deep reinforcement learning [60], [61].”
Benchmarks for Deep Off-Policy Evaluation (2021, background): “However, this sort of active online interaction is often impractical for real-world problems, where active data collection can be costly (Li et al., 2010), dangerous (Hauskrecht & Fraser, 2000; Kendall et al., 2019), or time consuming (Gu et al., 2017).”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, methodology): “For continuous control tasks, on-policy algorithms such as REINFORCE [2], TRPO [10], A3C [14], PPO [17] and off-policy algorithms such as DDPG [13] and SAC [21] have enabled successful applications such as quadrupedal locomotion [20], self-driving [30] or dexterous in-hand manipulation [20, 25, 32].”
Reward (Mis)design for Autonomous Driving (2021, background): “Lastly, we are confident that 4 papers do not include reward shaping [23,8,25,33].”
A review on Deep Reinforcement Learning for Fluid Mechanics (2019, background): “DRL has also proven itself useful in many industrial applications, such as autonomous cars [9, 10], or data center cooling [11].”
A Survey of State-Action Representations for Autonomous Driving (2018, methodology): “Finally, camera images being high-dimensional observations, compressed representations can be learnt explicitly through the use of unsupervised learning and auto-encoders, such as in (Ha and Schmidhuber, 2018; Kendall et al., 2018).”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, background): “, autonomous driving [98, 122, 144, 130, 133, 188, 189, 288], robotics [90, 92, 214, 216, 256, 272], video compression [177], vehicle schedule [22, 159], etc.”
Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks (2021, methodology): “• Driving as Markov Decision Process(MDP) [26]: The continuous learning of lane following on nonstationary data can be performed by using the model-”
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, background): “Kendall et al. [30] train their deep RL algorithm onboard to efﬁciently learn to drive a real-world vehicle.”
MASK-RL: Multiagent Video Object Segmentation Framework Through Reinforcement Learning (2020, background): “For these reasons, RL has recently attracted a lot of attention for emulating human behavior in several contexts, such as games [21], [29] and autonomous driving [22], but also for computer vision tasks [1], [13], [19], [25], [27], [47], including video object segmentation [11], [12], [37].”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, background): “PETR [31] introduces 3D positional encoding to image features and uses detection queries to learn object features via attention [46] mechanism.”
End-To-End Interpretable Neural Motion Planner (2019, methodology): “Following the success of Alpha GO [29], RL has been applied to self-driving in [15, 23].”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “Zeng et al. [75] deﬁne the cost volume responsible for safe planning; Kendall et al. [44] propose a practical safety reward function for safety-sensitive outputs.”
Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning (2020, background): “Two main branches for end-to-end autonomous driving are imitation learning (IL) [3], [4], [6], [7], which learns a driving policy by imitating the collected expert driving data, and reinforcement learning (RL) [8]–[10], which learns a policy by self exploration and reinforcement.”
Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning (2019, methodology): “Learning algorithms can be used to solve the tasks in an integrated/end-to-end fashion [13] [14] or in an isolated approach for each driving task like intersectiondriving [15] [16] and lane-changing [17].”