Mind's Eye: Grounded Language Model Reasoning through Simulation

By Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, Andrew M. Dai. 2022

103 citations recorded by the source corpus.

Successful and effective communication between humans and AI relies on a shared experience of the world. By training solely on written text, current language models (LMs) miss the grounded experience of humans in the real-world -- their failure to relate language to the physical world causes knowledge to be misrepresented and obvious mistakes in their reasoning. We present Mind's Eye, a paradigm to ground language model reasoning in the physical world. Given a physical reasoning question, we use a computational physics engine (DeepMind's MuJoCo) to simulate the possible outcomes, and then use the simulation results as part of the input, which enables language models to perform reasoning. Experiments on 39 tasks in a physics alignment benchmark demonstrate that Mind's Eye can improve reasoning ability by a large margin (27.9% zero-shot, and 46.0% few-shot absolute accuracy improvement on average). Smaller language models armed with Mind's Eye can obtain similar performance to models that are 100x larger. Finally, we confirm the robustness of Mind's Eye through ablation studies.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Tool Learning with Foundation Models (2023, unknown): “MIND’S EYE (Liu et al., 2022) Physical Simulation Engine Direct Concatenation in Model Input”
Cognitive Architectures for Language Agents (2023, background): “This includes interacting with games (Hausknecht et al., 2020; Côté et al., 2019; Shridhar et al., 2020; Wang et al., 2022a; Liu et al., 2023d), APIs (Schick et al., 2023; Yao et al., 2022b; Parisi et al., 2022; Tang et al., 2023b), and websites (Shi et al., 2017; Nakano et al., 2021; Yao et al.,…”
MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (2023, background): “For example, LLMs can connect with (1) the Internet to obtain latest information (Nakano et al., 2021; Shuster et al., 2022; Paranjape et al., 2023; Liu et al., 2023b); (2) the program interpreter to run the generated code (Chen et al., 2022; Gao et al., 2023; Drori et al., 2022; Pan et al., 2023;…”
Large Language Models as Tool Makers (2023, background): “…analogous to the evolution of human intelligence, recent research has unveiled the potential of augmenting LLMs with external tools , thereby significantly enhancing their problem-solving capacities and efficiencies (Yao et al., 2023; Liu et al., 2023; Parisi et al., 2022; Schick et al., 2023).”
When large language models meet personalization: perspectives of challenges and opportunities (2023, unknown): “Mind’s Eye[224] Physics Engine Text-to-code LM gpt-3 PaLM Reasoning”
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, background): “, 2023), incoherent logical ability (Liu et al., 2022; Ling et al., 2023) and arithmetic ability (Gao et al.”
Language Models can Solve Computer Tasks (2023, methodology): “Huang et al. [28] showed that LLMs can be used to plan and achieve simple household tasks, utilizing a method for grounding the actions generated by LLMs by comparing their embeddings with a predefined list of admissible actions.”
Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling (2023, methodology): “Other prior work used environment interactions to ground responses from question answering models in environment state (Gordon et al., 2018; Das et al., 2018) or physics (Liu et al., 2022).”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, background): “However, LLMs (and by extension VLMs) may lack the necessary grounding to perform well at social reasoning [40], physical reasoning [25], physics reasoning [44], embodied tasks [1, 32, 55], and spatial reasoning tasks [42, 52].”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, methodology): “We can teach the LLMs to use tools by pre-training [102], fine-tuning [93, 103], or in-context learning [97, 101].”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “Large language models have shown a wide range of capabilities: reasoning [19, 20], logic [21, 22], math [23], physics [24, 25], high-level planning [1, 26, 27, 28, 29, 30] with language feedback [31, 2], and writing robot code [5, 32, 33].”