Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation

By Lei Tai, Giuseppe Paolo, Ming Liu. 2017

807 citations recorded by the source corpus.

We present a learning-based mapless motion planner by taking the sparse 10-dimensional range findings and the target position with respect to the mobile robot coordinate frame as input and the continuous steering commands as output. Traditional motion planners for mobile ground robots with a laser range sensor mostly depend on the obstacle map of the navigation environment where both the highly precise laser sensor and the obstacle map building work of the environment are indispensable. We show that, through an asynchronous deep reinforcement learning method, a mapless motion planner can be trained end-to-end without any manually designed features and prior demonstrations. The trained planner can be directly applied in unseen virtual and real environments. The experiments show that the proposed mapless motion planner can navigate the nonholonomic mobile robot to the desired targets without colliding with any obstacles.

Subfields: Aerospace Engineering, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “Tai et al. (2017), in contrast, propose a learning-based mapless motion planner, under the assumption that the relative position of the target w.r.t. the robot can be obtained via cheap solutions such as wiﬁ or visible light localization, which are applicable to indoor robotic systems such as…”
The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation (2018, background): “…strong navigation performance in the indoor environments, the recently proposed deep reinforcement learning (DRL) based methods [1], [2], [3], [4] open up a new possibility that avoids the complexity of constructing these sophisticated SLAM pipelines and directly learns model-free policies…”
Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning (2017, background): “robot motion planning, they have mainly focused on a discrete action space [13], [21] or on small-scale problems [8], [9], [16], [17].”
One-Shot Reinforcement Learning for Robot Navigation with Interactive Replay (2017, background): “In robotics, agents have been trained to navigate using laser sensors in simulation [15] and simulated depth imagery [16]; structure-based sensors tend to exhibit less difference between simulation and reality (a narrower “reality gap”) than does visual sensing.”
HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators (2019, background): “In robotics, this success has mostly been restricted to either navigation [7, 8, 9, 10] or manipulation with stationary arms [2, 11, 12, 13, 14].”
A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontiers (2022, background): “This strategy is typical in deep reinforcement learning (DRL) approaches [121], [122], [123], for which local optimality is alleviated by network memorization.”
Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Mapless Navigation by Leveraging Prior Demonstrations (2018, methodology): “We also compare to models presented in prior work [3].”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “Early attempts aimed to verify RL’s capability in solving navigation problems typically solved with modular classical approaches (73).”
Socially Compliant Navigation Through Raw Depth Inputs with Generative Adversarial Imitation Learning (2017, background): “from raw sensory inputs [5], [6], [7], [8].”
Neural SLAM (2017, background): “Although an increasing amount of works have been done on utilizing deep reinforcement learning algorithms for autonomous navigation [13] [23] [22] [7] [19], none of them have an explicit external memory architecture to equip the agent with capabilities of making long-term decisions based on an inter”
Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems (2019, methodology): “[10] successfully appplied DRL for mapless navigation by taking the sqarse 10-dimensional range findings and the target position , defining mobile robot coordinate frame as input and continuous steering commands as output.”
Global Pose Estimation with an Attention-Based Recurrent Network (2018, background): “Targetdriven visual navigation takes a visual observation and an image of the target [50] or range findings [41] as input, and”
A Review of Domain Adaptation without Target Labels (2019, background): “In robotics, simulations can be employed as an additional source of data [49], [50].”
Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction (2019, neutral): “Significant progress has been made in many real-world applications, such as playing video games [17, 16, 4], robot control [6, 14], and robot navigation [12, 5, 32, 27].”
Lifelong Navigation (2020, methodology): “Learning-based Navigation Data-driven machine learning techniques have also been widely applied to navigation problems [10, 11, 12, 13, 14, 15, 16, 17, 18].”
Deep reinforcement learning based mobile robot navigation: A review (2021, methodology): “Generally, based on the target distance, moving closer to a target will generate a small positive reward [24, 34, 58, 60] , and moving closer to an obstacle will generate a small negative reward [69] .”
Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios (2020, methodology): “To overcome this limitation, Tai et al. (2017) proposed a map-less motion planner trained through a DRL method and Kahn et al. (2017) presented an uncertainty-aware model-based reinforcement-learning algorithm to estimate the probability of collision in an unknown environment.”