ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
By Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg. 2022
1,032 citations recorded by the source corpus.
Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information. However, such methods either require enumerating all possible next steps for scoring, or generate free-form text that may contain actions not possible on a given robot in its current context. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments, robot capabilities, and tasks. Our key insight is to prompt the LLM with program-like specifications of the available actions and objects in an environment, as well as with example programs that can be executed. We make concrete recommendations about prompt structure and generation constraints through ablation experiments, demonstrate state of the art success rates in VirtualHome household tasks, and deploy our method on a physical robot arm for tabletop tasks. Website at progprompt.github.io
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Chelsea Finn
- Percy Liang
- Quoc V. Le
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
- Li Fei-Fei
In Large Language Models
- Yejin Choi
- Graham Neubig
- Hannaneh Hajishirzi
- Sergey Levine
- Percy Liang
- Jifeng Dai
- Andy Zeng
- Noah D. Goodman
In Robotics & Control
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Corey Lynch
- Pierre Sermanet
- Yuke Zhu
- Andy Zeng
- Nicolas Manfred Otto Heess
Citations in the corpus
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “, 2023), program generation (Liang et al., 2022; Singh et al., 2022), or injecting information into the prompt (Zeng et al.”
- Visual Programming: Compositional visual reasoning without training (2022, background): “Similar to VISPROG, ProgPrompt [26] is a concurrent work that demonstrates the ability of LLMs to generate python-like situated robot action plans from natural language instructions.”
- VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, neutral): “To allow language models to perceive the physical environments, textual descriptions of the scene [39, 11, 59] or perception APIs [74] can be given, vision can be used during decoding [67] or can be directly taken as input by multi-modal language models [68, 2].”
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, positive): “…and vision foundation models to enhance robotic capabilities, utilizing them as pretrained visual representations that accelerate robotic policy learning [29, 32, 30, 19, 31], for object localization in robotics tasks [9, 45], and for high-level planning and reasoning [1, 17, 42, 16, 43, 18, 6].”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “Object detectors, VLMs, and control policies can be invoked via APIs, as shown in Figure 6b. ProgPrompt [171] introduces a novel task-planning approach by prompting LLMs with program-like specifications detailing available actions and objects.”
- A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “Most existing methods utilize prompting strategies for task planning (Singh et al., 2022; Zhou et al., 2023b; Song et al., 2023c; Wang et al., 2023g; Yao et al., 2023b; Liu et al., 2023j; Hao et al., 2023; Hu et al., 2023a), or building human-curated data for training (Lin et al., 2023a; Valmeekam…”
- Empowering Biomedical Discovery with AI Agents (2024, positive): “Then, since a single LLM can exhibit a broad range of capabilities—especially when configured with appropriate prompts and inference settings, conversations between differently configured agents can combine these capabilities in a modular manner [13].”
- Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory (2023, background): “Several works [4, 13, 21] have explored to augment LLMs with robot perception and control abilities.”
- Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling (2023, methodology): “, 2022b), or improving LLM plans through more structured output (Singh et al., 2022; Liang et al., 2022b).”
- Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “In ProgPrompt [41], the authors introduce a prompting method that uses LLMs to generate sequences of actions directly with no additional domain knowledge.”
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, background): “[62] decode robotic plans with pythonic text, and several works incorporate planning definition and domain language into the outputs [41, 61, 71].”
- A Comprehensive Overview of Large Language Models (2023, neutral): “action [176], [177], [178], [179], task planning [180], [181], [182], navigation [183], [184], and learning [185], [186].”
- A Survey on Multimodal Large Language Models for Autonomous Driving (2023, methodology): “Building on this, Liu et al. introduced LLaMa [98], which combines a vision encoder with an LLM to enhance the understanding of both visual and linguistic concepts.”
- LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, methodology): “Existing works use LLMs as a static generator for HLP [12, 13, 22, 30, 33, 40], use LLMs as a ranker that ranks all admissible actions in the current environment [1, 22], or add ground-truth environmental input to the LLM prompt [33].”
- Language Models of Code are Few-Shot Commonsense Learners (2022, background): “Existing works leverage the control-flows, nested functions and API calls of a programming language such as Python to control the situated agents in the embodied environment (Sun et al., 2019; Zhou et al., 2022; Singh et al., 2022).”
- Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “In the future, incorporating asserts at the execution level (e.g. checking whether the agent is close to the microwave, as in Singh et al. (2022)), is a promising research direction.”
- A survey on large language model based autonomous agents (2023, neutral): “In Progprompt [126], assertions are incorporated into the generated script to provide environment state feedback, allowing for error recovery in case the action’s preconditions are not satisfied.”
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “…difficulties, a line of recent works in building multimodal systems has leveraged pre-trained LLMs, with some augmenting with new modality encoders [5, 46, 119], some adding modality specific decoders [14, 96] and others leveraging the LLM’s capabilities to build modular frameworks [64, 166, 173].”
- Cognitive Architectures for Language Agents (2023, background): “Recent advances in LLMs have led to numerous robotic projects (Ahn et al., 2022; Liang et al., 2023a; Singh et al., 2023; Palo et al., 2023; Ren et al., 2023) that leverage LLMs as a “brain” for robots to generate actions or plans in the physical world.”
- Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents (2023, methodology): “Baselines We compare DEPS with other language-based planners, including GPT as Zero-shot Planner(GPT) [16], ProgPrompt(PP) [39], Chain-of-Thought(CoT) [41], Inner Monologue(IM) [17], and Code as Policies(CaP) [20].”