GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
By Iman Mirzadeh, Keivan Alizadeh-Vahid, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar. International Conference on Learning Representations, 2024
590 citations recorded by the source corpus.
Recent advancements in Large Language Models (LLMs) have sparked interest in their formal reasoning capabilities, particularly in mathematics. The GSM8K benchmark is widely used to assess the mathematical reasoning of models on grade-school-level questions. While the performance of LLMs on GSM8K has significantly improved in recent years, it remains unclear whether their mathematical reasoning capabilities have genuinely advanced, raising questions about the reliability of the reported metrics. To address these concerns, we conduct a large-scale study on several SOTA open and closed models. To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions. GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models.Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question. Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. Furthermore, we investigate the fragility of mathematical reasoning in these models and show that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is because current LLMs cannot perform genuine logical reasoning; they replicate reasoning steps from their training data. Adding a single clause that seems relevant to the question causes significant performance drops (up to 65%) across all state-of-the-art models, even though the clause doesn't contribute to the reasoning chain needed for the final answer. Overall, our work offers a more nuanced understanding of LLMs'capabilities and limitations in mathematical reasoning.
View Paper: https://www.semanticscholar.org/paper/05506581cade1a8ef6372616cec20b81a3d5c366
Citations in the corpus
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (2025, unknown): “Earlier iterations of LLMs [9, 10, 11] exhibited poor performance on reasoning benchmarks [12, 13, 14, 6].”
- Gemma 3 Technical Report (2025, unknown): “However, despite the use of decontamination techniques, there is always a risk of contamination of these probes (Mirzadeh et al., 2024), making more definitive conclusions harder to assess.”
- LIMO: Less is More for Reasoning (2025, unknown): “While this approach has achieved significant gains, it has been criticized for relying on memorization rather than true generalization (Mirzadeh et al., 2024; Zhang et al., 2024).”
- Interpretability Framework for LLMs in Undergraduate Calculus (2025, unknown): “The models often replicate reasoning steps from their training data rather than performing genuine logical reasoning [17].”
- CRANE: Reasoning with constrained LLM generation (2025, unknown): “…demonstrate that CRANE significantly outperforms both SOTA constrained decoding strategies and standard un-constrained decoding, showing up to a 10% improvement over baselines on challenging symbolic reasoning benchmarks GSM-symbolic (Mirzadeh et al., 2024)) and FO-LIO (Han et al., 2024).”