LILA: A Unified Benchmark for Mathematical Reasoning

By Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark and 1 more. 2022

171 citations recorded by the source corpus.

Mathematical reasoning skills are essential for general-purpose intelligentsystems to perform tasks from grocery shopping to climate modeling.Towards evaluating and improving AI systems in this domain, we proposeLILA, a unified mathematical reasoning benchmark consisting of 23 diversetasks along four dimensions:(i) mathematical abilities e.g., arithmetic, calculus (ii) language format e.g., question-answering, fill-in-the-blanks (iii) language diversity e.g., no language, simple language (iv) external knowledge e.g., commonsense, physics. We construct our benchmark by extending 20 datasets benchmark by collecting task instructions and solutions in the form of Python programs,thereby obtaining explainable solutions in addition to the correct answer.We additionally introduce two evaluation datasets to measure out-of-distribution performance and robustness to language perturbation.Finally, we introduce BHASKARA,a general-purpose mathematical reasoning model trained on LILA. Importantly, we find that multi-tasking leads to significant improvements (average relative improvement of 21.83% F1 score vs. single-task models),while the best performing model only obtains 60.40%,indicating the room for improvement in general mathematical reasoning and understanding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PAL: Program-aided Language Models (2022, background): “…a wide range of reasoning tasks, including commonsense (Wei et al., 2021; Sanh et al., 2021; Madaan et al., 2022), mathematical (Lewkowycz et al., 2022; Wu et al., 2022; Mishra et al., 2022), and symbolic reasoning (Yao et al., 2022; Ahn et al., 2022), using few-shot prompting (Brown et al., 2020).”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022, neutral): “Recent research work [4, 37, 33] has shown that GPT-3 [4] can perform various tasks when provided with in-context examples in a standard prompt.”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “The total number of training examples is 776K. • English mathematical datasets : We annotate GSM8K and MATH problems with tool-integrated solutions, and adopt a subset of MathInstruct (Yue et al., 2023) along with the training set of Lila-OOD (Mishra et al., 2022) where problems are solved with CoT ”
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, background): “LiLA also annotates Python programs as the generation target for solving mathematical problems.”
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, background): “Type Annotation # Samples Characteristics Fields GSM8K (Cobbe et al., 2021) CoT Human 7K Grade Schol Exam ∎ GSM8K-RFT (Yuan et al., 2023) CoT Llama 28K Llama + Validated ∎ AQuA-RAT (Ling et al., 2017) CoT Human 90K GRE/GMAT Exam ∎ MATH (Hendrycks et al., 2021b) CoT Human 7K Math Competition ∎ ∎ ∎ ∎ ”
Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement (2023, methodology): “For other datasets, we adopt the split from (Mishra et al., 2022) or randomly select 200 instances, and perform 3 restarts for stable results.”
Generating Sequences by Learning to Self-Correct (2022, methodology): “As in prior work (Austin et al., 2021; Ni et al., 2022; Mishra et al., 2022), we frame these as program synthesis by converting their solutions to Python programs.”
Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, methodology): “Mishra et al. (2022) conducted fine-tuning on a series of GPT-Neo2.”
OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text (2023, methodology): “This includes the subset of level-1 algebra questions from MATH, LILA-multiarith to test coding ability, and GSM8k and MATH perplexities, which scale more smoothly than accuracies.”
Reasoning with Language Model Prompting: A Survey (2022, background): “Most recently, Mishra et al. (2022a) extends existing datasets to construct a unified benchmark concerning mathematical abilities, language diversity, and external knowledge.”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “There is strong evidence that deep learning meth-ods for mathematical reasoning are not robust and susceptible to adversarial attacks (Lin et al., 2020; Patel et al., 2021; Mishra et al., 2022b,a; Welleck et al., 2022b).”
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, methodology): “Most existing approaches are either limited to a small number of tools [33, 6, 47, 15, 37, 43] or rely on domain-specific tools [34, 52, 11, 51, 45], and thus are not easy to generalize to diverse queries.”
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, methodology): “LILA [33] extends 20 datasets by including task instructions and Python program solutions.”
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (2023, methodology): “To overcome the limitations, recent research has exploited tools like calculators (Cobbe et al., 2021; Shao et al., 2022), code interpreters (Mishra et al., 2022), and symbolic solvers (Zhang et al., 2023).”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, result): “This is also consistent with previous findings (Fu et al., 2022; Mishra et al., 2022; Wei et al., 2022b).”