ProgPrompt: Generating Situated Robot Task Plans using Large Language Models

By Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg. 2022

1,032 citations recorded by the source corpus.

Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information. However, such methods either require enumerating all possible next steps for scoring, or generate free-form text that may contain actions not possible on a given robot in its current context. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments, robot capabilities, and tasks. Our key insight is to prompt the LLM with program-like specifications of the available actions and objects in an environment, as well as with example programs that can be executed. We make concrete recommendations about prompt structure and generation constraints through ablation experiments, demonstrate state of the art success rates in VirtualHome household tasks, and deploy our method on a physical robot arm for tabletop tasks. Website at progprompt.github.io

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Yoshua Bengio
  5. Chelsea Finn
  6. Percy Liang
  7. Quoc V. Le
  8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Jitendra Malik
  6. Ming-Hsuan Yang
  7. Andrew Zisserman
  8. Li Fei-Fei

In Large Language Models

  1. Yejin Choi
  2. Graham Neubig
  3. Hannaneh Hajishirzi
  4. Sergey Levine
  5. Percy Liang
  6. Jifeng Dai
  7. Andy Zeng
  8. Noah D. Goodman

In Robotics & Control

  1. Sergey Levine
  2. Chelsea Finn
  3. Pieter Abbeel
  4. Corey Lynch
  5. Pierre Sermanet
  6. Yuke Zhu
  7. Andy Zeng
  8. Nicolas Manfred Otto Heess

Citations in the corpus