HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators

By Chengshu Li, Fei Xia, Roberto Martin Martin, Silvio Savarese. 2019

123 citations recorded by the source corpus.

Most common navigation tasks in human environments require auxiliary arm interactions, e.g. opening doors, pressing buttons and pushing obstacles away. This type of navigation tasks, which we call Interactive Navigation, requires the use of mobile manipulators: mobile bases with manipulation capabilities. Interactive Navigation tasks are usually long-horizon and composed of heterogeneous phases of pure navigation, pure manipulation, and their combination. Using the wrong part of the embodiment is inefficient and hinders progress. We propose HRL4IN, a novel Hierarchical RL architecture for Interactive Navigation tasks. HRL4IN exploits the exploration benefits of HRL over flat RL for long-horizon tasks thanks to temporally extended commitments towards subgoals. Different from other HRL solutions, HRL4IN handles the heterogeneous nature of the Interactive Navigation task by creating subgoals in different spaces in different phases of the task. Moreover, HRL4IN selects different parts of the embodiment to use for each phase, improving energy efficiency. We evaluate HRL4IN against flat PPO and HAC, a state-of-the-art HRL algorithm, on Interactive Navigation in two environments - a 2D grid-world environment and a 3D environment with physics simulation. We show that HRL4IN significantly outperforms its baselines in terms of task performance and energy efficiency. More information is available at this https URL.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In LLM Agents & Tool Use

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Others learn to hierarchically solve such long-horizon problems [79, 80, 12, 81, 54].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Others have approached the problem through hierarchical learning [45, 46, 34, 47, 48, 49, 50].”
Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments (2019, methodology): “While in this letter we only use Interactive Gibson Environment to study Interactive Navigation, the environment opens up new avenues to study and train novel visual navigation, manipulation and mobile manipulation solutions exploiting the synergies between navigation and interaction [14].”
NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration (2023, background): “Prior works have often addressed this challenge by training a separate high-level policy or goal proposal system that generates suitable exploratory tasks, for example using high-level planning [1], hierarchical reinforcement learning [2], and generative models [3].”
iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, background): “learn to control physical interactions [1, 2], navigate based on sensor signals [3, 4, 5, 6, 7], or plan long horizon tasks [8, 9, 10, 11].”
ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation (2020, background): “This is potentially caused by our deviation from the original task setup in [17] since we do not allow collisions with the robot base during exploration.”
ManipulaTHOR: A Framework for Visual Object Manipulation (2021, background): “[22, 35] study mobile manipulation by generating sub-goal tasks.”
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, background): “…such a reference problem directly in 3D space – i.e., without a camera view dependency – can beneﬁt many downstream robotics applications, including embodied question answering [21], visual-and language-based navigation [9], instruction following [58], and manipulating objects in a scene [67,37].”
BEHAVIOR-1K: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation (2022, methodology): “To facilitate our experiments, we adopt an assistive pick primitive that creates a rigid connection between the object and the gripper if grasping is requested when all ﬁngers are in contact with the object, a stricter form of StickyMitten used in prior works [26, 63, 64].”
ReLMoGen: Integrating Motion Generation in Reinforcement Learning for Mobile Manipulation (2021, methodology): “Mobile Manipulation (MM) Tasks: These are long-horizon tasks known to be difﬁcult for RL [48, 17], making it a good test for our method.”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, methodology): “via reinforcement-learning [32, 26, 67, 37, 65, 20, 18, 11, 69], and recently through foundation models [3, 38, 66].”
The rise and potential of large language model based agents: a survey (2025, unknown): “In traditional hierarchical reinforcement learning (HRL) methods, the high-level policy constraints sub-goals for the low-level policy which produces appropriate action signals [263–265].”