Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
By Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, Yitao Liang. 2023
502 citations recorded by the source corpus.
We investigate the challenge of task planning for multi-task embodied agents in open-world environments. Two main difficulties are identified: 1) executing plans in an open-world environment (e.g., Minecraft) necessitates accurate and multi-step reasoning due to the long-term nature of tasks, and 2) as vanilla planners do not consider how easy the current agent can achieve a given sub-task when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient or even infeasible. To this end, we propose"$\underline{D}$escribe, $\underline{E}$xplain, $\underline{P}$lan and $\underline{S}$elect"($\textbf{DEPS}$), an interactive planning approach based on Large Language Models (LLMs). DEPS facilitates better error correction on initial LLM-generated $\textit{plan}$ by integrating $\textit{description}$ of the plan execution process and providing self-$\textit{explanation}$ of feedback when encountering failures during the extended planning phases. Furthermore, it includes a goal $\textit{selector}$, which is a trainable module that ranks parallel candidate sub-goals based on the estimated steps of completion, consequently refining the initial plan. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the $\texttt{ObtainDiamond}$ grand challenge with our approach. The code is released at https://github.com/CraftJarvis/MC-Planner.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Percy Liang
- Chelsea Finn
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
- Li Fei-Fei
In Large Language Models
In LLM Agents & Tool Use
- Brian Ichter
- Andy Zeng
- Stephen Marcus McAleer
- Pierre Baldi
- Geunwoo Kim
- Corey Lynch
- Tianmin Shu
- Pierre Sermanet
Citations in the corpus
- Augmented Language Models: a Survey (2023, methodology): “Along these lines, Wang et al. (2023) uses a LM as a centralized planner to generate goal sequences for solving tasks in the game of Minecraft.”
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “, 2022a), visual explanations (Wang et al., 2023), program generation (Liang et al.”
- WorldSimBench: Towards Video Generation Models as World Simulators (2024, methodology): “Previous works (Qin et al., 2024; Wang et al., 2023d;a) have primarily focused on exploring the capabilities of LLMs or MLLMs as Predictive Text Modelat the S 1 stage.”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “DEPS [175] stands for “Describe, Explain, Plan and Select”.”
- Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models (2023, background): “…et al., 2022b; Li et al., 2022; Weng et al., 2022; Fu et al., 2022), problem decomposition (Zhou et al., 2022; Khot et al., 2022; Dua et al., 2022; Press et al., 2022), and planning (Yao et al., 2022; Huang et al., 2022; Wang et al., 2023; Liu et al., 2023; Sun et al., 2023; Yao et al., 2023).”
- Building Cooperative Embodied Agents Modularly with Large Language Models (2023, result): “In contrast to the above, our work addresses a more challenging multi-agent cooperation problem, characterized by decentralized control, complex observations, costly communication , and long-horizon multi-objective tasks .”
- Tool Learning with Foundation Models (2023, unknown): “…reasoning additionally considers feedback from the user and environment (Figure 5), and is thus better suited to complex tasks (Madaan et al., 2023; Wang et al., 2023b), such as multi-step QA and embodied learning, where decision-making at each step is dependent on the preceding context.”
- Executable Code Actions Elicit Better LLM Agents (2024, background): “For example, it may suffer from hallucination commonly seen in LLMs (e.g., imagine the content of a variable without actually printing it out), suggesting the need for subsequent alignment (Ouyang et al., 2022) for further improvements.”
- A survey on large language model based autonomous agents (2023, unknown): “Based on the planning capability of LLMs, DEPS [154] develops a Minecraft”
- An Embodied Generalist Agent in 3D World (2023, unknown): “Ad-6 ditionally, we test generalization to the validation split of the newly introduced HM3D ObjNav task (Ramakrishnan et al., 2021).”
- Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory (2023, methodology): “DEPS [25] further extended this closed-loop interaction by introducing description,”
- Language Models can Solve Computer Tasks (2023, methodology): “First, our experiments use LLMs on HTML code, but ideally methods based on multimodal foundation models [13, 41, 2, 35] will be able to take actions based on text, images, audio, and video as input [3, 15, 33, 55].”
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models (2023, background): “As they are trained on vast amount of world knowledge and human examples, LMs are known to have already absorbed rich commonsense that makes it possible to propose reasonable plans conditioned on problem setting and environmental states [10, 39, 34, 11, 32, 38, 37].”
- Graph of Thoughts: Solving Elaborate Problems with Large Language Models (2023, background): “LLMs & Planning There are many works on how to plan complex tasks with LLMs (Huang et al. 2022a,b; Zhang et al. 2023; Yao et al. 2023b; Yang et al. 2023; Wang et al. 2023c).”
- The rise and potential of large language model based agents: a survey (2025, unknown): “…to control low-level policies [163] like a robotic transformer [101, 281, 282], which takes images and instructions as inputs and produces control commands for the end effector and robotic arms as well as virtual embodied controllers due to high costs of robotic operators [16,89,90,139,271,283,284].”
- Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, background): “Most recently, studies show that LLM can be a few-shot policy for task planning [18, 2, 19] in the large domain, while it may suffer from hallucinations.”