Visual Room Rearrangement

By Luca Weihs, Matt Deitke, Aniruddha Kembhavi, Roozbeh Mottaghi. 2021

157 citations recorded by the source corpus.

There has been a significant recent progress in the field of Embodied AI with researchers developing models and algorithms enabling embodied agents to navigate and interact within completely unseen environments. In this paper, we propose a new dataset and baseline models for the task of Rearrangement. We particularly focus on the task of Room Rearrangement: an agent begins by exploring a room and recording objects’ initial configurations. We then remove the agent and change the poses and states (e.g., open/closed) of some objects in the room. The agent must restore the initial configurations of all objects in the room. Our dataset, named RoomR, includes 6,000 distinct rearrangement settings involving 72 different object types in 120 scenes. Our experiments show that solving this challenging interactive task that involves navigation and object interaction is beyond the capabilities of the current state-of-the-art techniques for embodied tasks and we are still very far from achieving perfect performance on these types of tasks.

Subfields: Computer Vision and Pattern Recognition, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Robotics & Control

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “[58] Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi.”
Continuous Scene Representations for Embodied AI (2022, methodology): “• VRR: Corresponds to the (RN18, PPO+IL) from [52].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “Our action space of size |A| = 110 is the union of all navigation actions and valid object interactions following [62].”
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments (2021, unknown): “…several lines of pioneering efforts have been made towards benchmarking embodied AI in simulated environments, including Rearrangement [23, 24], TDW Transport Challenge [25], VirtualHome [26], ALFRED [11], Interactive Gibson Benchmark [27], MetaWorld [28], and RLBench [29], among others…”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, methodology): “We consider an architecture and reward structure similar to other work in embodied AI (e.g., [65, 36]).”
Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, background): “It has many practical implications in everyday life [3, 34, 38, 20, 17], such as setting the table, tidying up the room, loading the dishwasher, etc..”
Simple but Effective: CLIP Embeddings for Embodied AI (2021, methodology): “An instance of the ROOMR task has been developed for the simulated iTHOR environment [31].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “, 2022), rearrangement (Weihs et al., 2021; Szot et al., 2021; Liu et al., 2021; Ehsani et al., 2021; Gan et al., 2021; Stengel-Eskin et al., 2022), and reasoning (Gupta et al.”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, methodology): “We also consider two manipulation-based tasks: ArmPoint-Nav [16] and 1-phase Room Rearrangement [55].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Versions: [197], [198] Benchmarks: ALFRED, RoPOR [199] VirtualHome [200] 7”
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks (2021, unknown): “This kinodynamic simulation is sufficient for pick-and-place and rearrangement tasks [11, 12, 13, 14]; however, as the field advances, researchers are taking on more diverse and complex tasks that cannot be performed in these simulators, e.”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “…substantial attention, and notable advancements have been made in various tasks (Anderson et al., 2018; Krantz et al., 2020; Jain et al., 2019; Savva et al., 2019; Deitke et al., 2020; Weihs et al., 2021; Ehsani et al., 2021; Shridhar et al., 2020; Pad-makumar et al., 2022; Gao et al., 2022).”
BEHAVIOR-1K: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation (2022, background): “Unlike geometric, image/video, or experience goal speciﬁcations [17, 18], BDDL deﬁnitions are in terms of objects and object states, allowing annotators to deﬁne at an intuitive semantic level.”
TidyBot: Personalized Robot Assistance with Large Language Models (2023, methodology): “Below, we describe how we use the LLM to infer personalized rules for both receptacle selection and manipulation primitive selection, and also how we deploy the approach on a real-world mobile manipulation system for household cleanup.”