Habitat 2.0: Training Home Assistants to Rearrange their Habitat

By Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets and 11 more. 2021

769 citations recorded by the source corpus.

We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack - data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850x real-time) on an 8-GPU node, representing 100x speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from 'hand-off problems', and (3) SPA pipelines are more brittle than RL policies.

Subfields: Artificial Intelligence, Control and Systems Engineering, Mechanical Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Mechanical Engineering

In Robotics & Control

Citations in the corpus

BEHAVIOR-1K: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation (2022, methodology): “To facilitate our experiments, we adopt an assistive pick primitive that creates a rigid connection between the object and the gripper if grasping is requested when all ﬁngers are in contact with the object, a stricter form of StickyMitten used in prior works [26, 63, 64].”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, neutral): “That is, if one policy performs better than another in real-world 1] , higher is better) for assessing the correlation between policy performances in real-world and simulation, as well as the overall quality of simulated evaluation pipelines.”
Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks (2023, methodology): “Another 3D simulation library with a similar purpose is DeepMind Lab [2], but given that the game engine is written in C and the levels are written using Lua, there is a steep learning curve for customizing it.”
Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items (2022, methodology): “through manual 3D modeling [19], object scanning [20], conversion of CAD models [21], or combinations of these techniques [22].”
Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, background): “It has many practical implications in everyday life [3, 34, 38, 20, 17], such as setting the table, tidying up the room, loading the dishwasher, etc..”
RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024, methodology): “RoboCasa runs realistic physics for all interactions, including object grasping and placement, unlike some other mobile manipulation frameworks such as AI2-THOR [21] and Habitat 2.0 [40].”
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (2023, methodology): “We curate C ORTEX B ENCH , a benchmark for evaluating PVRs, consisting of 17 tasks spanning low-level locomotion [10], table-top manipulation [11], dexterous manipulation [12], multi-finger coordinated manipulation [13], indoor visual navigation [14], and mobile manipulation [15].”
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments (2023, background): “While recent frameworks [7], [11], [12] include physically-based renderers, they mainly support camera-based sensors (RGB, depth).”
Objaverse: A Universe of Annotated 3D Objects (2022, methodology): “To put the scale of O BJAVERSE -Interiors in perspective, the number of scenes in O BJAVERSE -Interiors is significantly larger than the 400 or so existing hand-built interactive embodied AI scenes [21,36,39,67].”
Efficient Continual Learning Ensembles in Neural Network Subspaces (2022, background): “…or mobile devices or wearable embodied AI as examples, where not only must the inference be carried on-device, but there is also a need to train an always-on AI model that continually learns while interacting with the user or perceiving the world (Manolis Savva* et al., 2019; Szot et al., 2021).”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, neutral): “A = EMEQA Agent(Q, H) , that is “sim-ilar” to the ground truth answer A ⇤ .”