Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

By Abulhair Saparov, He He. International Conference on Learning Representations, 2022

512 citations recorded by the source corpus.

Large language models (LLMs) have shown remarkable reasoning capabilities given chain-of-thought prompts (examples with intermediate reasoning steps). Existing benchmarks measure reasoning ability indirectly, by evaluating accuracy on downstream tasks such as mathematical reasoning. However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought. To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called PrOntoQA, where each example is generated from a synthetic world model represented in first-order logic. This allows us to parse the generated chain-of-thought into symbolic proofs for formal analysis. Our analysis on InstructGPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts. However, they have difficulty with proof planning: When multiple valid deduction steps are available, they are not able to systematically explore the different options.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/e7028cd7ea838ab8294ecf26d5a2c0dbb8cfa81a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2022, background): “Saparov and He (2023) investigates systematically evaluating CoT by creating a synthetic QA dataset based on first-”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, methodology): “…Question-Answering [52, 53], it observed that although the most extensive model exhibited capability in reasoning tasks, it encountered challenges in proof planning and the selection of the appropriate proof step amidst multiple options, which will cause the uncertainty of accuracy [52].”
A Survey for In-context Learning (2023, background): “Saparov and He (2022) generated an example from a synthetic world model represented in ﬁrst-order logic and parsed the ICL generations into symbolic proofs for formal analysis.”
Large Language Models are Better Reasoners with Self-Verification (2022, background): “This approach has been shown the advance performances on several challenging NLP tasks, even when using only a few or no training samples (Madaan et al., 2022; Saparov and He, 2022; Fu et al., 2022).”
LAMBADA: Backward Chaining for Automated Reasoning in Natural Language (2022, methodology): “This dataset has multiple variants; we use the fictional characters version, which is one of the hardest versions accordingn to the results in (Saparov and He, 2022).”
Towards Reasoning in Large Language Models: A Survey (2022, background): “Saparov and He (2022) create a synthetic dataset called PrOntoQA that is generated from real or fictional ontologies.”
Training Large Language Models to Reason in a Continuous Latent Space (2024, unknown): “We found that the generation process of ProntoQA could be more challenging, especially since the size of distracting branches in the ontology is always small, reducing the need for complex planning.”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, unknown): “To showcase the efficacy of tree-search algorithms across varied search spaces, we evaluate all algorithms on four tasks with different search widths and depths, including the mathematical reasoning task GSM8k (Cobbe et al., 2021), mathematical planning task Game24 (Yao et al., 2023), logical reason”
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, unknown): “Tasks designed to test for logical reasoning, whether deductive (Saparov & He, 2023, PrOntoQA), inductive (Bowen et al., 2024) or analogical (Ma et al., 2024) reasoning, including syllogisms and logical puzzles.”
Faith and Fate: Limits of Transformers on Compositionality (2023, background): “Challenges of Transformers in compositional tasks Transformers perform fairly well in singlestep reasoning tasks [52], but face challenges when it comes to effectively combining multiple steps to solve compositionally complex problems [38, 48, 62].”
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023, background): “, 2021), PrOntoQA (Saparov and He, 2023), FOLIO (Han et al.”
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models (2023, background): “Subsequently, many works (Wang et al., 2022a; Suzgun et al., 2022; Shaikh et al., 2022; Saparov and He, 2022) propose to further improve CoT prompting in different aspects, including prompt format (Chen et al., 2022), prompt selection (Lu et al., 2022), prompt ensemble (Wang et al., 2022b; Li et…”
Reasoning with Language Model is Planning with World Model (2023, methodology): “We assess the performance of our RAP framework on PrOntoQA (Saparov and He, 2022) and adopt their settings of “true” ontology (using real-world knowledge), “random” ordering of rules.”
A Survey of Large Language Models (2023, unknown): “Complex Reasoning Knowledge Reasoning CSQA [247], StrategyQA [248], ARC [291], BoolQ [315], PIQA [295], SIQA [316], HellaSwag [317], WinoGrande [318], OpenBookQA [302], COPA [319], ScienceQA [301], proScript [320], ProPara [321], ExplaGraphs [322], ProofWriter [323], EntailmentBank [324], ProOntoQA ”
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, background): “…(NLP) tasks such as summarization (Nallapati et al., 2016) and language inference (Bow-man et al., 2015), but they have also been adapted to an increasingly diverse set of tasks that often require advanced common-sense reasoning or quantitative skills (Cobbe et al., 2021; Saparov and He, 2023).”