Open-vocabulary Queryable Scene Representations for Real World Planning
By Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S. Ryoo, Austin Stone, Daniel Kappler. 2022
257 citations recorded by the source corpus.
Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable scene representation with Visual Language models (VLMs). An LLM based object proposal module parses instructions and proposes involved objects to query the scene representation for object availability and location. An LLM planner then plans with such information about the scene. NLMap allows robots to operate without a fixed list of objects nor executable options, enabling real robot operation unachievable by previous methods. Project website: https://nlmap-saycan.github.io.
Subfields: Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Multimodal Learning, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Andrew Zisserman
- Sergey Levine
- Li Fei-Fei
In LLM Agents & Tool Use
- Andy Zeng
- Sergey Levine
- Kevin Qinghong Lin
- Antonio Torralba
- Fei Xia
- Karthik Narasimhan
- Marco Pavone
- Christopher Agia
In Multimodal Learning
- Chuang Gan
- Josh Tenenbaum
- Antonio Torralba
- Karthik Narasimhan
- Sergey Levine
- Jasmine Hsu
- Kevin Qinghong Lin
- Shuang Li
In Robotics & Control
Citations in the corpus
- ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “…the 2D representations produced by image and language foundation models to the 3D world and show impressive results on open-vocabulary tasks, including language-guided object grounding [17], [18], [24], [26], [36], 3D reasoning [37], [38], robot manipulation [39]–[41] and navigation [42], [43].”
- VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “Leveraging pre-trained language models for embodied applications is an active area of research, where a large body of works focus on planning and reasoning with language models [9–11, 58, 31, 39, 59–72, 36, 73, 74].”
- Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, unknown): “Moving into the domain of real-world planning with large language models, Chen et al. (2023b) introduce NLMap, which creates an open-vocabulary and queryable scene representation, allowing language models to gather and integrate contextual information for context-conditioned planning.”
- ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “VLMaps [44], LM-Nav [45], CoWs [46], and NLMap-Saycan [47] leverage the open-set features from pixel-aligned LSeg [24] models for robot navigation based on language commands.”
- Text2Motion: from natural language instructions to feasible plans (2023, methodology): “Once the Q-functions have converged, we distill their returns into stochastic policies π through the maximum-entropy update [74]: Instead of evaluating the policies on D v , which contains states for which no feasible action exists, the policies are synchronously evaluated in an environment that…”
- RT-1: Robotics Transformer for Real-World Control at Scale (2022, methodology): “via reinforcement-learning [32, 26, 67, 37, 65, 20, 18, 11, 69], and recently through foundation models [3, 38, 66].”
- LangSplat: 3D Language Gaussian Splatting (2023, background): “The field of open-ended language queries in 3D has attracted increasing attention due to its various applications such as robotic navigation [14] and manipulation [35], 3D semantic understanding [9, 49] and editing [20], autonomous driving [16], and augmented/virtual reality [23].”
- OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, background): “Recent success of 2D open-vocabulary segmentation models such as LSeg [43], OpenSeg [20], and OV-Seg[45] has motivated researchers in the field of 3D scene understanding to explore the open vocabulary setting [6, 12, 19, 24, 30, 32, 35, 37, 49, 52, 59, 60].”
- Cognitive Architectures for Language Agents (2023, methodology): “Finally, work in robotics (Chen et al., 2023a) uses vision-language models to build a semantic map of the environment, which can later be queried to execute instructions.”
- Visual Language Maps for Robot Navigation (2022, background): “The recent success of large pretrained vision and language models [10], [27] has spurred a flurry of interest in applying their zero-sot capabilities to different domains including object detection and segmentation [28], [29], [11], robot manipulation [30], [31], [32], [33], and navigation [13], [12”
- LERF: Language Embedded Radiance Fields (2023, methodology): “CLIP-Fields [32] and NLMaps-SayCan [8] fuse CLIP embeddings of crops into pointclouds, using a contrastively supervised field and classical pointcloud fusion respectively.”
- TidyBot: Personalized Robot Assistance with Large Language Models (2023, methodology): “, 2022), scene description as context (Zeng et al., 2022; Mees et al., 2022; Chen et al., 2022; Singh et al., 2022),”