Learning to Drive from Simulation without Real World Labels

By Alex Bewley, Jessica Rigley, Jeffrey Hawke, Richard Shen, Vinh-Dieu Lam, Alex Kendall. 2018

118 citations recorded by the source corpus.

Simulation can be a powerful tool for under-standing machine learning systems and designing methods to solve real-world problems. Training and evaluating methods purely in simulation is often “doomed to succeed” at the desired task in a simulated environment, but the resulting models are incapable of operation in the real world. Here we present and evaluate a method for transferring a vision-based lane following driving policy from simulation to operation on a rural road without any real-world labels. Our approach leverages recent advances in image-to-image translation to achieve domain transfer while jointly learning a single-camera control policy from simulation control labels. We assess the driving performance of this method using both open-loop regression metrics, and closed-loop performance operating an autonomous vehicle on rural and urban roads.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, background): “Recently, the trend of sim2real [3] also aims to bridge the gap between synthetic and real data.”
Habitat: A Platform for Embodied AI Research (2019, background): “Once a promising approach has been developed and tested in simulation, it can be transferred to physical platforms that operate in the real world [6, 15].”
Learning by Cheating (2019, background): “They can be transferred to the physical world using any procedure for sim-to-real transfer [4, 14].”
nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles (2021, background): “[3] learn how to drive by transferring a vision-based lane following driving policy from simulation to the real world without any real-world labels.”
Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving (2019, methodology): “Pixel level [37], GraspGAN [38], [39], [40] Object Detection SeGAN [41], [42], Perceptual-GAN [43] Super Resolution SRGAN [23] Inpainting [44], [45], [46] Advanced Data Augmentation Context-aware Synthesis and Placement of Object Instances [20] domains without supervision.”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, methodology): “The described method [78] was tested on a real vehicle and outperformed other domain adaptation methods, including simulation-to-real [46] and realto-simulation [77] transformation.”
A review on Deep Reinforcement Learning for Fluid Mechanics (2019, background): “domains such as robotics [5], language processing [6], or games [7, 8], but DRL has also proven useful 26 in many industrial applications, such as autonomous cars [9, 10], or data center cooling [11].”
Synthetic Data for Deep Learning (2019, background): “[55] perform domain adaptation for learning to drive from simulation; they claim to present the first end-to-end driving policy transferred from an (obviously supervised) synthetic setting to the fully unsupervised real domain.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Typically, domain-invariant feature learning is achieved with image translators and discriminators to map images from two domains into a common latent space or representations like seg-mentationmaps [262], [263].”