Virtual to Real Reinforcement Learning for Autonomous Driving

By Yurong You, Xinlei Pan, Ziyan Wang, Cewu Lu. 2017

356 citations recorded by the source corpus.

Reinforcement learning is considered as a promising direction for driving policy learning. However, training autonomous driving vehicle with reinforcement learning in real environment involves non-affordable trial-and-error. It is more desirable to first train in a virtual environment and then transfer to the real environment. In this paper, we propose a novel realistic translation network to make model trained in virtual environment be workable in real world. The proposed network can convert non-realistic virtual image input into a realistic one with similar scene structure. Given realistic frames as input, driving policy trained by reinforcement learning can nicely adapt to real world driving. Experiments show that our proposed virtual to real (VR) reinforcement learning (RL) works pretty well. To our knowledge, this is the first successful case of driving policy trained by reinforcement learning that can adapt to real world driving data.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “…al., 2017) Depth/Color/Semantics 20s+FPS Autonomous Driving Carla (Dosovitskiy et al., 2017) Depth/Color/Semantics 30s+FPS Autonomous Driving Torcs (You et al., 2017) Color/Semantics 100s+FPS Autonomous Driving AI2-Thor (Kolve et al., 2017) Color 100s+FPS Indoor Navigation Minos (Savva et al.,…”
Driving Policy Transfer via Modularity and Abstraction (2018, background): “complementary approach that uses modularity to encapsulate the policy and abstract some of the nuisance factors that were tackled explicitly by Sadeghi and Levine via domain randomization. Pan et al. [34] use generative adversarial networks to adapt driving models from simulation to real images, and”
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning (2018, background): “Other approaches have sought to map datapoints from one domain to another [48, 59, 5, 60].”
Advantage Actor-Critic Methods for CarRacing (2018, methodology): “…discretize continuous control in one of the following two ways: discretize each factor individually and retain the conditional independence assumption as done in the OpenAI-5 Dota Bolt (openAI, 2018) or manually assemble a finite set of policy combinations from expert knowledge (You et al., 2017).”
Planning and Decision-Making for Autonomous Vehicles (2018, methodology): “The gap between simulation and real-world data could be closed (82) by first segmenting the virtual image from the simulator with a segmentation network and then translating it into a realistic-looking image employing a generative network.”
Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving (2019, methodology): “Reinforcement Learning for Autonomous Driving model trained in virtual environment is shown to perform well in real environment [39].”
End to End Vehicle Lateral Control Using a Single Fisheye Camera (2018, unknown): “[2] tried to go one step closer to real test by first training a Generative Adversarial Network [3] to generate real-looking images from the synthetic images of the simulation and then give these generated images as input to the RL algorithm.”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, background): “in [9]), one might also just explicitly punish crashes by assigning a large negative reward, as was done in [26], [61], [66], [69], [70].”
Unsupervised Real-to-Virtual Domain Unification for End-to-End Highway Driving (2018, background): “[34] trains a driving policy with reinforcement learning in a simulator by transforming virtual images to real images, retaining the scene structure with adversarial loss on the segmentation mask.”
Virtual-to-Real: Learning to Control in Visual Semantic Segmentation (2018, background): “As the image semantics rendered by the simulators and those generated by the perception module are invariant [You et al., 2017], the control policy learned from the simulators can be transferred to the real world directly.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Typically, domain-invariant feature learning is achieved with image translators and discriminators to map images from two domains into a common latent space or representations like seg-mentationmaps [262], [263].”
An Introduction to Deep Reinforcement Learning (2018, unknown): “They may provide robust solutions to some of the self-driving cars challenges (Bojarski et al., 2016; You et al., 2017).”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, background): “DA could be useful for AIoT applications when deploying models to new scenarios or new working modes of machines [128], [203]–[206], e.”
Synthetic Data for Deep Learning (2019, unknown): “he source and target domains and learn to nd matching synthetic images to produce aligned data. The resulting model is successfully applied to training a visuomotor policy for real robots. Pan et al. [676] consider sim-to-real translation for autonomous driving; they convert synthetic images to a sc”
A Survey of State-Action Representations for Autonomous Driving (2018, background): “Another objective of preprocessing can also be the transfer of skills learnt in simulation to real use cases, as Pan et al. (2017) who use style transfer to convert the simulator images distribution to a realistic images distribution that is fed to the driving policy during its training, so as to…”