Continuous Scene Representations for Embodied AI

By S. Gadre, Kiana Ehsani, Shuran Song, Roozbeh Mottaghi. 2022

62 citations recorded by the source corpus.

We propose Continuous Scene Representations (CSR), a scene representation constructed by an embodied agent navigating within a space, where objects and their relationships are modeled by continuous valued embeddings. Our method captures feature relationships between objects, composes them into a graph structure on-the-fly, and situates an embodied agent within the representation. Our key insight is to embed pair-wise relationships between objects in a latent space. This allows for a richer representation compared to discrete relations (e.g., [SUPPORT], [NEXT-TO]) commonly used for building scene representations. CSR can track objects as the agent moves in a scene, update the representation accordingly, and detect changes in room configurations. Using CSR, we outperform state-of-the-art approaches for the challenging downstream task of visual room rearrangement, without any task specific training. Moreover, we show the learned embeddings capture salient spatial details of the scene and show applicability to real world data. A summery video and code is available at prior.allenai.org/projects/csr.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Robotics & Control

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Of important, the study of general continual learning helps the embodied agents to learn from an egocentric perception similar to humans, and provides a unique opportunity for researchers to pursue the essence of lifelong learning by observing the same person in a long time span [124], [425].”
Simple but Effective: CLIP Embeddings for Embodied AI (2021, methodology): “We use this data for all tasks except reachability, for which we instead use the dataset and annotations from [11].”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, background): “Some researchers also propose supervised rewards to encourage exploration [65, 24].”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, background): “40 [36] Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song, and Roozbeh Mottaghi.”