Open-vocabulary Queryable Scene Representations for Real World Planning

By Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S. Ryoo, Austin Stone, Daniel Kappler. 2022

257 citations recorded by the source corpus.

Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable scene representation with Visual Language models (VLMs). An LLM based object proposal module parses instructions and proposes involved objects to query the scene representation for object availability and location. An LLM planner then plans with such information about the scene. NLMap allows robots to operate without a fixed list of objects nor executable options, enabling real robot operation unachievable by previous methods. Project website: https://nlmap-saycan.github.io.

Subfields: Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Multimodal Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Jitendra Malik
  3. Dacheng Tao
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Andrew Zisserman
  7. Sergey Levine
  8. Li Fei-Fei

In LLM Agents & Tool Use

  1. Andy Zeng
  2. Sergey Levine
  3. Kevin Qinghong Lin
  4. Antonio Torralba
  5. Fei Xia
  6. Karthik Narasimhan
  7. Marco Pavone
  8. Christopher Agia

In Multimodal Learning

  1. Chuang Gan
  2. Josh Tenenbaum
  3. Antonio Torralba
  4. Karthik Narasimhan
  5. Sergey Levine
  6. Jasmine Hsu
  7. Kevin Qinghong Lin
  8. Shuang Li

In Robotics & Control

  1. Sergey Levine
  2. Chelsea Finn
  3. Andy Zeng
  4. Tianhe Yu
  5. Julian Ibarz
  6. Fei Xia
  7. Shuran Song
  8. Brian Ichter

Citations in the corpus