ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

By Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi and 1 more. Advances in Neural Information Processing Systems 35, 2022

516 citations recorded by the source corpus.

Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding. This work presents a platform to enable similar success stories in Embodied AI. We propose ProcTHOR, a framework for procedural generation of Embodied AI environments. ProcTHOR enables us to sample arbitrarily large datasets of diverse, interactive, customizable, and performant virtual environments to train and evaluate embodied agents across navigation, interaction, and manipulation tasks. We demonstrate the power and potential of ProcTHOR via a sample of 10,000 generated houses and a simple neural model. Models trained using only RGB images on ProcTHOR, with no explicit mapping and no human task supervision produce state-of-the-art results across 6 embodied AI benchmarks for navigation, rearrangement, and arm manipulation, including the presently running Habitat 2022, AI2-THOR Rearrangement 2022, and RoboTHOR challenges. We also demonstrate strong 0-shot results on these benchmarks, via pre-training on ProcTHOR with no fine-tuning on the downstream benchmark, often beating previous state-of-the-art systems that access the downstream training data.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/d7bca917b753b996074203cc77ba45935b7c62d7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Indeed, in the past few years a series of challenging PCG environments have been proposed (Juliani et al., 2019; Cobbe et al., 2020; Küttler et al., 2020; Samvelyan et al., 2021; Chevalier-Boisvert et al., 2018; Hafner, 2022; Deitke et al., 2022), mostly focusing on testing and improving generalisat”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “AI2-THOR (Ehsani et al., 2021; Deitke et al., 2022) is a framework that supports visual object manipulation and procedural generation of environments.”
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (2023, background): “Thesolutions,basedonautoregressivelanguagemodelinginGPT[47,48,4]and maskedautoencoding in BERT[14],areconceptuallysimple:theyremoveaportionofthedataandlearntopredicttheremovedcontent.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “Restrictions apply.”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, methodology): “We showcase this with a 3-DoF ( x , y and yaw) localization and remapping task in the AI2Thor [64], [65] simulation environment, where a mobile robot uses a particle filter to localize in a pre-built ConceptGraphs map of the environment.”
Objaverse: A Universe of Annotated 3D Objects (2022, methodology): “To facilitate this task, we pro-cedurally generate 10K new homes in ProcTHOR [16] fully populated with objects from O BJAVERSE -LVIS.”
SQA3D: Situated Question Answering in 3D Scenes (2022, unknown): “…tasks including robotic navigation (Das et al., 2018; Anderson et al., 2018; Savva et al., 2019; Chen et al., 2019; Wijmans et al., 2019b; Qi et al., 2020; Deitke et al., 2022) and vision-based manipulation (Kolve et al., 2017; Puig et al., 2018; Xie et al., 2019; Shridhar et al., 2020a;b; 2022).”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, background): “…exploration, many navigation tasks are goal-conditioned, where the agent needs to navigate to a speciﬁed position (i.e., point goal [11, 12, 26, 31, 66, 77, 81, 83]), view of the environment (i.e., image goal [48, 64, 89]), or object category (i.e., object goal [1, 9, 10, 12, 19, 44, 74, 79, 84]).”
eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents (2026, unknown): “• We introduce eMEM-Bench v1 , an embodied-memory benchmark built over ProcTHOR-10K scenes [Deitke et al., 2022] and organised explicitly around eight cognitive-psychology paradigms drawn from human-memory research — DRM lures [Roediger and McDermott, 1995], pattern separation [Yassa and Stark,…”