Reasoning with Language Model is Planning with World Model

By Shibo Hao, Yi Gu, Haodi Ma, J. Hong, Zhen Wang, Daisy Zhe Wang, Zhiting Hu. Conference on Empirical Methods in Natural Language Processing, 2023

1,077 citations recorded by the source corpus.

Large language models (LLMs) have shown remarkable reasoning capabilities, especially when prompted to generate intermediate reasoning steps (e.g., Chain-of-Thought, CoT). However, LLMs can still struggle with problems that are easy for humans, such as generating action plans for executing tasks in a given environment, or performing complex math, logical, and commonsense reasoning. The deficiency stems from the key fact that LLMs lack an internal $\textit{world model}$ to predict the world $\textit{state}$ (e.g., environment status, intermediate variable values) and simulate long-term outcomes of actions. This prevents LLMs from performing deliberate planning akin to human brains, which involves exploring alternative reasoning paths, anticipating future states and rewards, and iteratively refining existing reasoning steps. To overcome the limitations, we propose a new LLM reasoning framework, $\underline{R}$easoning vi$\underline{a}$ $\underline{P}$lanning $\textbf{(RAP)}$. RAP repurposes the LLM as both a world model and a reasoning agent, and incorporates a principled planning algorithm (based on Monto Carlo Tree Search) for strategic exploration in the vast reasoning space. During reasoning, the LLM (as agent) incrementally builds a reasoning tree under the guidance of the LLM (as world model) and task-specific rewards, and obtains a high-reward reasoning path efficiently with a proper balance between exploration $\textit{vs.}$ exploitation. We apply RAP to a variety of challenging reasoning problems including plan generation, math reasoning, and logical inference. Empirical results on these tasks demonstrate the superiority of RAP over various strong baselines, including CoT and least-to-most prompting with self-consistency. RAP on LLAMA-33B surpasses CoT on GPT-4 with 33% relative improvement in a plan generation setting.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/5dbffedcabe3fa43060ebbe2b1789500edfd871f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Cognitive Architectures for Language Agents (2023, methodology): “Depending on the form of action values, selection may occur via argmax, softmax, or an alternative such as majority vote (Wang et al., 2022b).”
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation (2023, methodology): “…generation methods that are based on LLMs but with various optimisation strategies, including Few-shot learning, Chain-of-Thought [Wei et al. , 2022 ] , ReAct [ Yao et al. , 2022 ] , Reflexion [ Shinn et al. , 2023], ToT [Yao et al. , 2023], RAP [Hao et al. , 2023], Self-Edit [Zhang et al. , 2023b”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “RAP [79] repurposes an LLM to act as both a policy that predicts actions and a world model that provides the state transition distribution.”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, neutral): “This is the setting used in ToT and RAP.”
Large Language Models as Analogical Reasoners (2023, background): “…with more structured algorithms and search methods (Khot et al., 2022; Drozdov et al., 2022; Zelikman et al., 2022; Yao et al., 2023; Press et al., 2022; Khattab et al., 2022; Jung et al., 2022), as well as longer-horizon action and planning (Yao et al., 2022; Hao et al., 2023; Park et al., 2023).”
Training Large Language Models to Reason in a Continuous Latent Space (2024, neutral): “While CoT has proven effective for certain tasks, its autoregressive generation nature makes it challenging to mimic human reasoning on more complex problems (LeCun, 2022; Hao et al., 2023), which typically require planning and search.”
LIMO: Less is More for Reasoning (2025, positive): “This can be achieved by augmenting LLMs with methods such as parallel sampling (Brown et al., 2024; Wang et al., 2022; Li et al., 2022) or symbolic tree search (Hao et al., 2023; Chen et al., 2024; Yao et al., 2023) to enhance reasoning ability.”
ExpeL: LLM Agents Are Experiential Learners (2023, methodology): “On the other hand, prompting methods can augment an LLM with better sequential decision-making planning abilities with only a few in-context examples (Hao et al. 2023; Lin et al. 2023b; Sun et al. 2023).”
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, background): “While this problem could be formalized with some kind of predicate logic (Zhou et al., 2022; Quan et al., 2024; Zhou et al., 2024) or grounded in some kind of physical simulation (Hao et al., 2023; Wong et al., 2023), there is not a natural nor well agreed-upon framework for solving it.”
A survey on large language model based autonomous agents (2023, methodology): “RAP [56] builds a world model to simulate the potential benefits of di ff erent plans based on Monte Carlo Tree Search (MCTS), and then, the final plan is generated by aggregating multiple MCTS iterations.”
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, methodology): “Tree-of-thought (ToT) prompting (Yao et al., 2023a) uses DFS or BFS-based (depth/breadth-first) search guided by an LM-generated heuristic, while reasoning via planning (RAP) (Hao et al., 2023) uses MCTS with rollouts simulated by LMs.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Recent methods [37, 44, 80, 85, 86] also induce language-based world models from large language models, but are restricted in textual space and struggle with grounding on physics [26, 59].”
Empowering Biomedical Discovery with AI Agents (2024, neutral): “To identify the optimal path, methods such as voting strategies [167], Monte Carlo tree search [169], and breadth/depth-ﬁrst search algorithms [164] are used.”
Improve Mathematical Reasoning in Language Models by Automated Process Supervision (2024, neutral): “Prompting LLMs with CoT examples (Wei et al., 2022b) and fine-tuning them on question-CoT solution pairs (Ouyang et al., 2022; Perez et al., 2021) have proven effective, with the latter demonstrating superior performance.”
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025, neutral): “…have been proposed [Kang et al., 2024, Wang et al., 2024a], including random sampling [Wang et al., 2023] and tree-search methods [Yao et al., 2024, Hao et al., 2023, Zhang et al., 2024b, Qi et al., 2024] Reward Models are crucial for effective System 2 reasoning but are challenging to obtain.”
Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, methodology): “On the other hand, similar to our work, some studies try to use LLMs as heuristics [42] or transition function [13] in MCTS, boosting the performance in coding or small-scale reasoning.”
Tree of Thoughts: Deliberate Problem Solving with Large Language Models (2023, methodology): “On the other hand, traditional decision-making procedures usually require training dedicated reward and policy models as in reinforcement learning (for example CHAI [30]), whereas we use the LM itself to provide the value estimates for decision making.”