On the Paradox of Learning to Reason from Data
By Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, Guy Van den Broeck. 2022
145 citations recorded by the source corpus.
Logical reasoning is needed in a wide range of NLP tasks. Can a BERT model be trained end-to-end to solve logical reasoning problems presented in natural language? We attempt to answer this question in a confined problem space where there exists a set of parameters that perfectly simulates logical reasoning. We make observations that seem to contradict each other: BERT attains near-perfect accuracy on in-distribution test examples while failing to generalize to other data distributions over the exact same problem space. Our study provides an explanation for this paradox: instead of learning to emulate the correct reasoning function, BERT has, in fact, learned statistical features that inherently exist in logical reasoning problems. We also show that it is infeasible to jointly remove statistical features from data, illustrating the difficulty of learning to reason in general. Our result naturally extends to other neural models (e.g. T5) and unveils the fundamental difference between learning to reason and learning to achieve high performance on NLP benchmarks using statistical features.
Subfields: Artificial Intelligence, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Murray Shanahan
- Antonia Creswell
- Sarath Sreedharan
- Subbarao Kambhampati
- He He
- Najoung Kim
- Greg Durrett
- Swarat Chaudhuri
In Prompting & Reasoning
- Murray Shanahan
- He He
- Kaj Bostrom
- Greg Durrett
- Zayne Sprague
- Abulhair Saparov
- Antonia Creswell
- Swarat Chaudhuri
Citations in the corpus
- Faithful Reasoning Using Large Language Models (2022, background): “While contemporary language models (LMs) are good at many natural language tasks, they often struggle with logical reasoning (Betz et al., 2021; Creswell et al., 2022; Dasgupta et al., 2022; Rae et al., 2021; Zhang et al., 2022).”
- LAMBADA: Backward Chaining for Automated Reasoning in Natural Language (2022, background): “…(Clark et al., 2020; Tafjord et al., 2020), further exploration revealed that finetuned LMs mostly exploit spurious correlations (e.g., correlation between the number of rules and the final conclusion) as opposed to learning to reason (Zhang et al., 2022b; Schlegel et al., 2022; Liu et al., 2022).”
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024, unknown): “…reasoning process in LLMs is fragile and not formal (Kambhampati, 2024; Valmeekam et al., 2022; 2024; Nezhurina et al., 2024; McCoy et al., 2023; Zhang et al., 2023), even though it appears that these models understand symbols and can work with them to some limited degree (Boix-Adserà et al.,…”
- On the Planning Abilities of Large Language Models - A Critical Investigation (2023, background): “[34] Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck.”
- Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought (2022, background): “Zhang et al. (2022b) showed that BERT is not able to learn to reason robustly, but they did not use CoT prompting and it is not obvious if their results generalize to LLMs, which we evaluate.”
- Natural Language Deduction through Search over Statement Compositions (2022, result): “…the computational overhead of sequence-to-sequence models places a hard limit on the tree size and premise count that can be handled in the end-to-end setting; moreover, recent results call into question how well end-to-end Transformers can generalize this type of reasoning (Zhang et al., 2022).”