A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution

By Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, Yoav Artzi. 2021

170 citations recorded by the source corpus.

Natural language provides an accessible and expressive interface to specify long-term tasks for robotic agents. However, non-experts are likely to specify such tasks with high-level instructions, which abstract over specific robot actions through several layers of abstraction. We propose that key to bridging this gap between language and robot actions over long execution horizons are persistent representations. We propose a persistent spatial semantic representation method, and show how it enables building an agent that performs hierarchical reasoning to effectively execute long-term tasks. We evaluate our approach on the ALFRED benchmark and achieve state-of-the-art results, despite completely avoiding the commonly used step-by-step instructions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Chelsea Finn
3. Pieter Abbeel
4. Trevor Darrell
5. Mohit Bansal
6. Ruslan Salakhutdinov
7. Yoshua Bengio
8. Abhinav Kumar Gupta

In Computer Vision and Pattern Recognition

1. Sergey Levine
2. Luc Van Gool
3. Trevor Darrell
4. Li Fei-Fei
5. Dacheng Tao
6. Jitendra Malik
7. Abhinav Kumar Gupta
8. Andrew Zisserman

In Natural Language Processing

1. Mohit Bansal
2. Antonio Torralba
3. Jacob Andreas
4. Ruslan Salakhutdinov
5. Mohit Shridhar
6. Yonatan Bisk
7. Dieter Fox
8. Sergey Levine

In LLM Agents & Tool Use

1. Lucas Manuelli
2. Dieter Fox
3. Mohit Shridhar
4. Brian Ichter
5. Chelsea Finn
6. Sergey Levine
7. Yonatan Bisk
8. Antonio Torralba

Citations in the corpus

• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “One approach to grounding language models in interaction is by learning downstream networks with pre-trained LLM representations [38, 22, 21, 39, 40, 41, 42, 43].”
• Skill Induction and Planning with Latent Language (2021, methodology): “To further improve (SL) 3 ’s navigation capabilities, we introduce two model variants in which sub-task specifications beginning Find. . . are executed by a either a planner with ground-truth environment information or a specialized navigation module from the HLSM model (Blukis et al., 2021) rather t”
• FILM: Following Instructions in Language with Modular Methods (2021, methodology): “Egocentric RGB is first processed into depth map and instance segmentation, with MaskRCNN (He et al., 2017) (and its implementation by Shridhar et al. (2021)) and the depth prediction method of Blukis et al. (2021); details of the training are explained in Section 5 .”
• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Like in object detection [83, 84] and vision-language grounding [85, 86], voxel maps have been used to build persistent scene representations.”
• LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, methodology): “We compare with two main baseline models, HLSM [3] and FILM [24].”
• A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Real : SayCan setup LEO [159] ConvNeXt [160], PointNet++ [161] Vicuna-7B Concat MLP Yes Sim-Mani : CLIPort task subset; Sim-Navi : Habitat-web 3D-LLM [57] CLIP, 3D-CLR [162], ConceptFusion [163] Flamingo, BLIP-2”