LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models

By Chan Hee Song, Jiaman Wu, Clay Washington, Brian M. Sadler, Wei-Lun Chao, Yu Su. IEEE International Conference on Computer Vision, 2022

743 citations recorded by the source corpus.

This study focuses on using large language models (LLMs) as a planner for embodied agents that can follow natural language instructions to complete complex tasks in a visually-perceived environment. The high data cost and poor sample efficiency of existing methods hinders the development of versatile agents that are capable of many tasks and can learn new tasks quickly. In this work, we propose a novel method, LLM-Planner, that harnesses the power of large language models to do few-shot planning for embodied agents. We further propose a simple but effective way to enhance LLMs with physical grounding to generate and update plans that are grounded in the current environment. Experiments on the ALFRED dataset show that our method can achieve very competitive few-shot performance: Despite using less than 0.5% of paired training data, LLM-Planner achieves competitive performance with recent baselines that are trained using the full training data. Existing methods can barely complete any task successfully under the same few-shot setting. Our work opens the door for developing versatile and sample-efficient embodied agents that can quickly learn many tasks. 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8ee45aeb7c97e3346cc62f216f673b91277ac718

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A survey on large language model based autonomous agents (2023, positive): “Previous work has demonstrated that LLMs can be used as decent planers to decompose complex task into simpler ones [45].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “LLM-Planner [166] introduces a novel approach to constructing a hierarchical policy comprising a high-level planner and a low-level planner.”
Mind2Web: Towards a Generalist Agent for the Web (2023, background): “Our setting is also connected to embodied AI, where an agent, guided by language instructions, carries out tasks in a physical environment [2, 28, 29].”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, background): “Other works have aimed to improve the planning performance of LLMs through prompt engineering [60] or collecting various forms of feedback such as sensory information [51, 20, 34], human corrections [60], self-corrections [46] or information of unmet preconditions [42, 56].”
Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling (2023, methodology): “However, task specific textual knowledge is expensive to obtain, prompting the use of web queries (Nottingham et al., 2022) or models pretrained on general world knowledge (Dambekodi et al., 2020; Suglia et al., 2021; Ichter et al., 2022; Huang et al., 2022b; Song et al., 2022b).”
GPT-Driver: Learning to Drive with GPT (2023, background): “5’s capability for detailed numerical reasoning in motion planning.”
A Comprehensive Overview of Large Language Models (2023, neutral): “robotics [196], [197], [198], [199], offering significant potential to enhance a robot’s ability to navigate complex environments with precision and adaptability.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “1) Action Utilizing APIs: A common approach involves providing LLMs with deﬁnitions of well-trained policy models to understand and use them effectively for speciﬁc tasks [186], [195].”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, positive): “…and vision foundation models to enhance robotic capabilities, utilizing them as pretrained visual representations that accelerate robotic policy learning [29, 32, 30, 19, 31], for object localization in robotics tasks [9, 45], and for high-level planning and reasoning [1, 17, 42, 16, 43, 18, 6].”
Empowering Biomedical Discovery with AI Agents (2024, positive): “Through direct reasoning, agents can generate multiple threads of thought that could consider the best pathways, protein targets, and experiments that can be run to test the role of a candidate protein target (Figure 5c).”
Building Cooperative Embodied Agents Modularly with Large Language Models (2023, background): “Recent research has also demonstrated that LLMs can drive embodied agents for single-agent tasks through zero-shot prompting for instruction following tasks (Huang et al., 2022a) or few-shot prompting for more complex long-horizon tasks (Song et al., 2022).”
Large Language Model Alignment: A Survey (2023, background): “, 2022), planning (Huang et al., 2022; Song et al., 2022), tool learning (Qin et al.”
Towards Reasoning in Large Language Models: A Survey (2022, methodology): “such as problem solving, decision making, and planning (Huang et al., 2022e,f; Song et al., 2022).”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “The principle behind smoothing is that any text used to model a language captures only a limited subset of its linguistic patterns (or terms, in this case).”