Rearrangement: A Challenge for Embodied AI

By Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi and 2 more. 2020

245 citations recorded by the source corpus.

We describe a framework for research and evaluation in Embodied AI. Our proposal is based on a canonical task: Rearrangement. A standard task can focus the development of new techniques and serve as a source of trained models that can be transferred to other settings. In the rearrangement task, the goal is to bring a given physical environment into a specified state. The goal state can be specified by object poses, by images, by a description in language, or by letting the agent experience the environment in the goal state. We characterize rearrangement scenarios along different axes and describe metrics for benchmarking rearrangement performance. To facilitate research and exploration, we present experimental testbeds of rearrangement scenarios in four different simulation environments. We anticipate that other datasets will be released and new simulation platforms will be built to support training of rearrangement agents and their deployment on physical systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Robotics & Control

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, methodology): “a ‘pick up X’ action) or a ‘magic pointer’ [10].”
Continuous Scene Representations for Embodied AI (2022, methodology): “We support experimentally that (1) without any task speciﬁc training , a simple planning approach that employs CSR as the underlying representation, outperforms a map-based, reinforcement and imitation learning baseline trained directly on the task of room rearrangement [4, 52].”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “Substantial modeling (Majumdar et al., 2020) and benchmark (Qi et al., 2020b) efforts in EAI navigation focus on identifying object landmarks (Blukis et al., 2018) and destinations (Batra et al., 2020b).”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “For example, Rearrangement (Batra et al., 2020), a type of Visual Goal , can be formulated as “Please re-arrange objects to match this { scene image } ”; Few-shot Imitation can embed video snippet in the prompt “Follow this motion trajectory for the wooden cube: { frame 1 } , { frame 2 } , { frame 3”
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI (2021, unknown): “Rearrangement of objects in an environment has recently been proposed as a potential test bed for embodied AI [3].”
TidyBot: Personalized Robot Assistance with Large Language Models (2023, methodology): “Below, we describe how we use the LLM to infer personalized rules for both receptacle selection and manipulation primitive selection, and also how we deploy the approach on a real-world mobile manipulation system for household cleanup.”
Simple but Effective: CLIP Embeddings for Embodied AI (2021, background): “A consortium of researchers in Embodied AI have recently proposed for Room Rearrangement to be the next frontier challenge [2] for existing models and methods.”
Building Cooperative Embodied Agents Modularly with Large Language Models (2023, methodology): “We instantiate the problem with two decentralized intelligent embodied agents (including humans) cooperating to accomplish a long-horizon rearrangement task (Batra et al., 2020) in an indoor multi-room environment.”
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, unknown): “We expect this trend will hold for tasks that rely even more on visual recognition, like ObjectGoal navigation [19], question answering [13], object rearrangement [50], etc .”
Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, background): “It is a challenging task [3] as the robot must navigate, locate target objects and positions, and execute multi-step planning.”
Visual Room Rearrangement (2021, methodology): “We address an instantiation of the rearrangement problem, an interactive task, recently introduced by Batra et al. [3].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “The pursuit of such agents has driven exciting new research in visual semantic planning [68], instruction following [58], and object rearrangement [3, 29], typically supported by advanced simulators [45, 32, 57, 22] where policies may be learned quickly and safely before potentially transferring to…”
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2020, background): “Some rearrangement tasks [8] may also be specified by a target goal observation og , in”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, background): “We also note that recent work using AI2-THOR has done evaluations with object manipulation tasks (Shridhar et al., 2020; Batra et al., 2020; Ehsani et al., 2021), paving the way for more complex action spaces.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “Inspired by the success in NLP, embodied agent research [29, 11, 94, 23] has seen a surge in adoption of the large-scale pre-training paradigm.”
Pushing it out of the Way: Interactive Visual Navigation (2021, background): “Our task is closer to existing tasks that consider navigation among movable obstacles [4, 18, 23, 31, 32, 43, 44].”