LLM+P: Empowering Large Language Models with Optimal Planning Proficiency

By Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qian Liu, Shiqi Zhang, Joydeep Biswas, Peter Stone. arXiv.org, 2023

653 citations recorded by the source corpus.

Large language models (LLMs) have demonstrated remarkable zero-shot generalization abilities: state-of-the-art chatbots can provide plausible answers to many common questions that arise in daily life. However, so far, LLMs cannot reliably solve long-horizon planning problems. By contrast, classical planners, once a problem is given in a formatted way, can use efficient search algorithms to quickly identify correct, or even optimal, plans. In an effort to get the best of both worlds, this paper introduces LLM+P, the first framework that incorporates the strengths of classical planners into LLMs. LLM+P takes in a natural language description of a planning problem, then returns a correct (or optimal) plan for solving that problem in natural language. LLM+P does so by first converting the language description into a file written in the planning domain definition language (PDDL), then leveraging classical planners to quickly find a solution, and then translating the found solution back into natural language. Along with LLM+P, we define a diverse set of different benchmark problems taken from common planning scenarios. Via a comprehensive set of experiments on these benchmark problems, we find that LLM+P is able to provide optimal solutions for most problems, while LLMs fail to provide even feasible plans for most problems.\footnote{The code and results are publicly available at https://github.com/Cranial-XIX/llm-pddl.git.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/003ef1cd670d01af05afa0d3c72d72228f494432

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023, background): “…with access to external tools and resources, such as the information retriever (Nakano et al., 2021; Shi et al., 2023; Lazaridou et al., 2022), calculator (Cobbe et al., 2021), code interpreter (Wang et al., 2022), planner (Liu et al., 2023a), and other pre-trained models (Shen et al., 2023).”
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models (2023, background): “…et al., 2022b; Li et al., 2022; Weng et al., 2022; Fu et al., 2022), problem decomposition (Zhou et al., 2022; Khot et al., 2022; Dua et al., 2022; Press et al., 2022), and planning (Yao et al., 2022; Huang et al., 2022; Wang et al., 2023; Liu et al., 2023; Sun et al., 2023; Yao et al., 2023).”
On the Planning Abilities of Large Language Models - A Critical Investigation (2023, background): “There have also been efforts which mostly depended on LLMs as “translators" of natural language problem/goal specification into formal specifications, which are then thrown over to sound external planners [38, 21].”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, methodology): “Besides, the goal specification can be obtained by using an LLM to parse the user’s command and convert it into a symbolic form, as done previously in [30, 58, 32].”
The rise and potential of large language model based agents: a survey (2025, positive): “Similarly, LLMs have shown the emergent ability of planning in forms of goal reformulation [80, 108], task decomposition [79, 109], and adjusting plans in response to environmental changes [81,110].”
A survey on large language model based autonomous agents (2023, neutral): “For a comprehensive overview, please refer to the right portion of Figure 5.”
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, background): “Other systems include first-order logic (Saparov & He, 2023; Hua et al., 2024) or planning languages (Liu et al., 2023a; Valmeekam et al., 2023).”
A Survey on Vision--Language--Action Models for Embodied AI (2024, critical): “LLM-DM [77] uses an LLM to construct world models in planning domain definition language (PDDL)—a capability that LLM+P [78] did not achieve, as its PDDL world model was hand-crafted.”
TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024, background): “The emergence of language agents powered by LLMs has further intensified discussions around their planning capabilities (Liu et al., 2023a; Valmeekam et al., 2023).”
AgentBench: Evaluating LLMs as Agents (2023, background): “Later, a bunch of advanced reasoning strategies [40; 70; 88; 46; 98; 27] and applications [59; 64; 51; 1] for LLM-as-Agent have emerged and arouse much public interest.”
Cognitive Architectures for Language Agents (2023, methodology): “More sophisticated agents use if-else or while-if code structures (Wang et al., 2023a; Park et al., 2023); while agents deployed in well-defined domains may utilize structured simulators (Haslum et al., 2019) to generate plausible rollouts (Liu et al., 2023a; Dagan et al., 2023).”
Empowering Biomedical Discovery with AI Agents (2024, neutral): “We classify reasoning modules into two categories: direct reasoning and reasoning with feedback, depending on whether agents adjust their plan in response to experimental or human feedback.”
Eureka: Human-Level Reward Design via Coding Large Language Models (2023, neutral): “…2023) to generate grounded and structured programmatic output for decision making and robotics problems (Liang et al., 2023; Singh et al., 2023; Wang et al., 2023b; Huang et al., 2023; Wang et al., 2023a; Liu et al., 2023; Silver et al., 2023; Ding et al., 2023; Lin et al., 2023; Xie et al., 2023).”