Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents

By Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, Yitao Liang. 2023

502 citations recorded by the source corpus.

We investigate the challenge of task planning for multi-task embodied agents in open-world environments. Two main difficulties are identified: 1) executing plans in an open-world environment (e.g., Minecraft) necessitates accurate and multi-step reasoning due to the long-term nature of tasks, and 2) as vanilla planners do not consider how easy the current agent can achieve a given sub-task when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient or even infeasible. To this end, we propose"$\underline{D}$escribe, $\underline{E}$xplain, $\underline{P}$lan and $\underline{S}$elect"($\textbf{DEPS}$), an interactive planning approach based on Large Language Models (LLMs). DEPS facilitates better error correction on initial LLM-generated $\textit{plan}$ by integrating $\textit{description}$ of the plan execution process and providing self-$\textit{explanation}$ of feedback when encountering failures during the extended planning phases. Furthermore, it includes a goal $\textit{selector}$, which is a trainable module that ranks parallel candidate sub-goals based on the estimated steps of completion, consequently refining the initial plan. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the $\texttt{ObtainDiamond}$ grand challenge with our approach. The code is released at https://github.com/CraftJarvis/MC-Planner.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Yoshua Bengio
  5. Quoc V. Le
  6. Percy Liang
  7. Chelsea Finn
  8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Jitendra Malik
  6. Ming-Hsuan Yang
  7. Andrew Zisserman
  8. Li Fei-Fei

In Large Language Models

  1. Sergey Levine
  2. Timo Schick
  3. Asli Celikyilmaz
  4. Andy Zeng
  5. Brian Ichter
  6. Hao Tian
  7. Jifeng Dai
  8. Yu Qiao

In LLM Agents & Tool Use

  1. Brian Ichter
  2. Andy Zeng
  3. Stephen Marcus McAleer
  4. Pierre Baldi
  5. Geunwoo Kim
  6. Corey Lynch
  7. Tianmin Shu
  8. Pierre Sermanet

Citations in the corpus