ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation

By Fei Xia, Chengshu Li, Roberto Mart'in-Mart'in, Or Litany, Alexander Toshev, Silvio Savarese. 2020

87 citations recorded by the source corpus.

Many Reinforcement Learning (RL) approaches use joint control signals (positions, velocities, torques) as action space for continuous control tasks. We propose to lift the action space to a higher level in the form of subgoals for a motion generator (a combination of motion planner and trajectory executor). We argue that, by lifting the action space and by leveraging sampling-based motion planners, we can efficiently use RL to solve complex, long-horizon tasks that could not be solved with existing RL methods in the original action space. We propose ReLMoGen -- a framework that combines a learned policy to predict subgoals and a motion generator to plan and execute the motion needed to reach these subgoals. To validate our method, we apply ReLMoGen to two types of tasks: 1) Interactive Navigation tasks, navigation problems where interactions with the environment are required to reach the destination, and 2) Mobile Manipulation tasks, manipulation tasks that require moving the robot base. These problems are challenging because they are usually long-horizon, hard to explore during training, and comprise alternating phases of navigation and interaction. Our method is benchmarked on a diverse set of seven robotics tasks in photo-realistic simulation environments. In all settings, ReLMoGen outperforms state-of-the-art Reinforcement Learning and Hierarchical Reinforcement Learning baselines. ReLMoGen also shows outstanding transferability between different motion generators at test time, indicating a great potential to transfer to real robots.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, methodology): “[20] develop hierarchical methods for mobile manipulation, combining RL policies for goal-generation and motion-planning for executing them.”
iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, background): “In these environments, agents learn to engage in physical interactions [1, 2], navigate based on sensor signals [3, 4, 5, 6, 7], or plan long-horizon tasks [8, 9, 10, 11].”
ManipulaTHOR: A Framework for Visual Object Manipulation (2021, background): “[22, 35] study mobile manipulation by generating sub-goal tasks.”
Language to Rewards for Robotic Skill Synthesis (2023, background): “However, in many previous works in reinforcement learning (RL) or model predictive control (MPC), manual reward design requires extensive domain expertise [17, 18, 19].”
BEHAVIOR-1K: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation (2022, background): “This supports the idea that some form of action-space abstraction must be necessary to solve long-horizon activities of BEHAVIOR-1K, as others reported [27, 26, 69].”
Pushing it out of the Way: Interactive Visual Navigation (2021, background): “Our task is closer to existing tasks that consider navigation among movable obstacles [4, 18, 23, 31, 32, 43, 44].”
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments (2021, unknown): “Previous works have demonstrated good results developing solutions in iGibson 2.0 that could transfer to real world [2, 85], and evaluated the similarities between simulated and real-world sensor signals [22].”
iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, background): “The output of the policy are points to interact (push) that are given to one of our integrated motion planners to generate an arm motion [10].”
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI (2021, unknown): “…3D virtual environments with both photo-realistic rendering and high-ﬁdelity physics simulations ( e.g . iGibson [46], TDW [15] and SAPIEN [48]). iGibson [46, 45] made initial progress in this area, introducing an interactive navigation task with partial observations in a 3D cluttered environment.”
Visual Room Rearrangement (2021, background): “There is a general trend towards interactive tasks [50, 41, 49].”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, methodology): “via reinforcement-learning [32, 26, 67, 37, 65, 20, 18, 11, 69], and recently through foundation models [3, 38, 66].”