PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-Based Planning

By Aleksandra Faust, Oscar Ramirez, Marek Fiser, Kenneth Oslund, Anthony G. Francis, James Davidson, Lydia Tapia. 2017

332 citations recorded by the source corpus.

We present PRM-RL, a hierarchical method for long-range navigation task completion that combines sampling-based path planning with reinforcement learning (RL). The RL agents learn short-range, point-to-point navigation policies that capture robot dynamics and task constraints without knowledge of the large-scale topology. Next, the sampling-based planners provide roadmaps which connect robot configurations that can be successfully navigated by the RL agent. The same RL agents are used to control the robot under the direction of the planning, enabling long-range navigation. We use the Probabilistic Roadmaps (PRMs) for the sampling-based planner. The RL agents are constructed using feature-based and deep neural net policies in continuous state and action spaces. We evaluate PRM-RL, both in simulation and on-robot, on two navigation tasks with non-trivial robot dynamics: end-to-end differential drive indoor navigation in office environments, and aerial cargo delivery in urban environments with load displacement constraints. Our results show improvement in task completion over both RL agents on their own and traditional sampling-based planners. In the indoor navigation task, PRM-RL successfully completes up to 215 m long trajectories under noisy sensor conditions, and the aerial cargo delivery completes flights over 1000 m without violating the task constraints in an environment 63 million times larger than used in training.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, methodology): “More recent work has however begun utilizing learning and transitioning towards hybrid methods, for example learning distributions for sampling [90], using reinforcement [98] or differentiating through the optimization [99].”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, background): “Some prior works have also combined planning algorithms with RL (Chiang et al., 2019; Faust et al., 2018; Savinov et al., 2018a), ﬁnding that the combination yields agents adept at reaching distant goals.”
Path Planning for Autonomous Mobile Robots: A Review (2021, methodology): “[158] combined RL with the Probabilistic Roadmap Method (PRM), which is one of the algorithms detailed next.”
Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation (2021, background): “Currently, deep RL has demonstrated great potential on complex tasks, from learning to play video games directly from pixels [4], [5] to making immediate decisions on robot behavior from camera inputs [6]–[8].”
Combining Optimal Control and Learning for Visual Navigation in Novel Environments (2019, background): “Several works also use end-to-end learning for navigation using laser scans [12, 13, 14], for training and combining a local planner with a higher level roadmap for long range navigation [15, 16, 17, 18], or visual servoing [19].”
HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators (2019, background): “In robotics, this success has mostly been restricted to either navigation [7, 8, 9, 10] or manipulation with stationary arms [2, 11, 12, 13, 14].”
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, background): “While there are promising successes in applying RL algorithms directly on real systems ([7], [8], [9], [10], [11]), their performance on more complex systems remains bottle-necked by the relative data inefﬁciency of RL algorithms.”
Learning a Decentralized Multi-arm Motion Planner (2020, background): “Others have explored the application of reinforcement learning to motion planing [20, 21, 22, 23, 24].”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, background): “This also allows the embedding of policies in classical motion planning structures [23, 24, 25, 26, 27, 28, 29, 30].”
Lyapunov-based Safe Policy Optimization for Continuous Control (2019, background): “2016), manipulation (Levine et al., 2016; Kalashnikov et al., 2018), navigation (Faust et al., 2018), and obstacle avoidance (Chiang et al., 2019).”
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action (2022, background): “Data-driven approaches to vision-based mobile robot navigation often use photorealistic simulators [46–49] or supervised data collection [50] to learn goal-reaching policies directly from raw observations.”
Differentiable Gaussian Process Motion Planning (2019, methodology): “Machine learning has been used to accelerate motion planning by combining reinforcement learning (RL) with sampling based planning [5], learning cost functions from demonstration [6], learning efﬁcient heuristics [7], and learning collision checking policies [8].”
Learning Object-conditioned Exploration using Distributed Soft Actor Critic (2020, methodology): “In more detail, control policies have been trained using supervised learning [26] and DDPG [27] for point navigation.”
Visual Representations for Semantic Target Driven Navigation (2018, background): “Given the model of the environment, state-of-the art planning and control strategies can be employed [11].”
Deep reinforcement learning based mobile robot navigation: A review (2021, methodology): “In this case, DRL requires additional global information provided by the traditional navigation technique [21] .”
Learning Navigation Behaviors End-to-End With AutoRL (2018, methodology): “Practical robot navigation typically uses a layered navigation stack [26]: given an occupancy map generated by a mapping algorithm such as SLAM [7], a global planner such as PRM [22] or RRT [24] finds a near-optimal path that a local planner (e.g., [14]) executes.”
NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration (2023, methodology): “Inspired by prior work [1, 11, 12], we factorize the exploration problem into (i) learned control policies that can take diverse, short-horizon actions, and (ii) a high-level planner based on a topological graph that uses the policy for long-horizon goal-seeking.”