Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning

By Liangming Pan, Alon Albalak, Xinyi Wang, William Yang Wang. Conference on Empirical Methods in Natural Language Processing, 2023

591 citations recorded by the source corpus.

Large Language Models (LLMs) have shown human-like reasoning abilities but still struggle with complex logical problems. This paper introduces a novel framework, Logic-LM, which integrates LLMs with symbolic solvers to improve logical problem-solving. Our method first utilizes LLMs to translate a natural language problem into a symbolic formulation. Afterward, a deterministic symbolic solver performs inference on the formulated problem. We also introduce a self-refinement module, which utilizes the symbolic solver's error messages to revise symbolic formalizations. We demonstrate Logic-LM's effectiveness on five logical reasoning datasets: ProofWriter, PrOntoQA, FOLIO, LogicalDeduction, and AR-LSAT. On average, Logic-LM achieves a significant performance boost of 39.2% over using LLM alone with standard prompting and 18.4% over LLM with chain-of-thought prompting. Our findings suggest that Logic-LM, by combining LLMs with symbolic logic, offers a promising avenue for faithful logical reasoning. Code and data are publicly available at https://github.com/teacherpeterpan/Logic-LLM.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/9e9e4df2996bac794c4f04cb887df3e553bae4fd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024, unknown): “To minimize the impact of potential errors in semantic parsing, we use the logical forms provided by Logic-LM.”
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, methodology): “Many symbolic and semi-symbolic tasks be broken down into two stages (Ye et al., 2023; Pan et al., 2023; Jiang et al., 2024): planning, either via a formal or informal specification via prompting (Sun et al., 2024; Wang et al., 2023b), and execution, using the same LM or external solvers.”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, methodology): “To address this limitation, frameworks have been developed to enable LLMs to utilize external tools for performing sub-tasks like arithmetic [45] and logical reasoning [39, 57].”
ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (2025, unknown): “Recent works have explored prompting and supervised fine-tuning methods [2, 14] to equip LLMs with tool-use capabilities.”
CRANE: Reasoning with constrained LLM generation (2025, unknown): “…systems that interact with traditional software tools like Python interpreters (OpenAI, 2024; Chen et al., 2023) for code generation (Suresh et al., 2024a;b; Jiang et al., 2024), logical solvers (Pan et al., 2023; Olaus-* son et al., 2023), and theorem provers (Wu et al., 2022; Yang et al., 2023).”