On the Planning Abilities of Large Language Models - A Critical Investigation

By Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, Subbarao Kambhampati. Neural Information Processing Systems, 2023

455 citations recorded by the source corpus.

Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) the effectiveness of LLMs in generating plans autonomously in commonsense planning tasks and (2) the potential of LLMs as a source of heuristic guidance for other agents (AI planners) in their planning tasks. We conduct a systematic study by generating a suite of instances on domains similar to the ones employed in the International Planning Competition and evaluate LLMs in two distinct modes: autonomous and heuristic. Our findings reveal that LLMs' ability to generate executable plans autonomously is rather limited, with the best model (GPT-4) having an average success rate of ~12% across the domains. However, the results in the heuristic mode show more promise. In the heuristic mode, we demonstrate that LLM-generated plans can improve the search process for underlying sound planners and additionally show that external verifiers can help provide feedback on the generated plans and back-prompt the LLM for better plan generation.

Subfields: Artificial Intelligence, Language and Linguistics

View Paper: https://www.semanticscholar.org/paper/dedfe929d182cc3537a9ed765d589b4735ce062a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “PlanBench [219], [220] comprehensively assesses various aspects of task planning ability, such as cost optimality, plan verification, and replanning.”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…methods utilize prompting strategies for task planning (Singh et al., 2022; Zhou et al., 2023b; Song et al., 2023c; Wang et al., 2023g; Yao et al., 2023b; Liu et al., 2023j; Hao et al., 2023; Hu et al., 2023a), or building human-curated data for training (Lin et al., 2023a; Valmeekam et al., 2023).”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, methodology): “Despite the improvements in planning performance, LLMs are still far from being a usable and reliable planner due to various factors: (a) LLMs have not yet demonstrated sufficient capabilities in reasoning and planning [24, 55, 53, 54, 31].”
PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change (2022, background): “These roles range from generating plans [11, 33] and heuristics [33, 1] to extracting planning knowledge [7].”
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, unknown): “Other systems include first-order logic (Saparov & He, 2023; Hua et al., 2024) or planning languages (Liu et al., 2023a; Valmeekam et al., 2023).”