EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference

By Abhilasha Ravichander, Aakanksha Naik, Carolyn Penstein Rosé, Eduard H. Hovy. 2019

87 citations recorded by the source corpus.

Quantitative reasoning is a higher-order reasoning skill that any intelligent natural language understanding system can reasonably be expected to handle. We present EQUATE (Evaluating Quantitative Understanding Aptitude in Textual Entailment), a new framework for quantitative reasoning in textual entailment. We benchmark the performance of 9 published NLI models on EQUATE, and find that on average, state-of-the-art methods do not achieve an absolute improvement over a majority-class baseline, suggesting that they do not implicitly learn to reason with quantities. We establish a new baseline Q-REAS that manipulates quantities symbolically. In comparison to the best performing NLI model, it achieves success on numerical reasoning tests (+24.2 %), but has limited verbal reasoning capabilities (-8.1 %). We hope our evaluation framework will support the development of models of quantitative reasoning in language understanding.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Reasoning Like Program Executors (2022, methodology): “EQUATE Table 10 presents performance break-down by subsets of EQUATE (Ravichander et al., 2019), where we compare P O E T -SQL BART and P O E T -SQL RoBERTa with their vanilla versions and previous baselines.”
Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension (2019, background): “Other recent work has shown that even state-of-the-art neural models struggle with numerical operations and quantitative reasoning when trained in an end-to-end manner (Saxton et al., 2019; Ravichander et al., 2019).”
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019, background): “Recent datasets such as DROP (Dua et al., 2019), EQUATE (Ravichander et al., 2019), or Mathe-matics Questions (Saxton et al., 2019) test numerical reasoning; they contain examples which require comparing, sorting, and adding numbers in natural language (e.g., Figure 2).”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, methodology): “Datasets such as DROP (Dua et al., 2019), EQUATE (Ravichander et al., 2019), or Mathematics Questions (Saxton et al., 2018) test numerical reasoning; they contain examples that require com-2 paring, sorting, and performing other complex mathematical tasks.”
Representing Numbers in NLP: a Survey and a Vision (2021, background): “Quantity Entailment tasks (Ravichander et al., 2019; Roy et al., 2015), analogous to Natural”
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022, background): “Same is true for majority of EQUATE data.”
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks (2020, background): “[ Dehaene, 2011; Ravichander et al. , 2019; Frank et al. , 2008 ] .”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “, 2019), EQUATE (Ravichander et al., 2019), or Mathematics Questions (Saxton et al.”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “EQUATE (Ravichander et al., 2019) formalizes aspects of quantitative reasoning in a natural language inference framework.”
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2020, background): “2) External challenge tests (Naik et al., 2018; Isabelle et al., 2017; Glockner et al., 2018; Ravichander et al., 2019; McCoy et al., 2019), where examples are constructed, either through automatic methods or by experts, exercising a specific phenomenon in isolation.”
ANLIzing the Adversarial Natural Language Inference Dataset (2020, background): “amples where numerical reasoning is crucial for determining the correct label, and break down into CARDINAL, ORDINAL—along the lines of Ravichander et al. (2019)—COUNTING and NOMINAL; the first two break down further into”
Robustness Gym: Unifying the NLP Evaluation Landscape (2021, unknown): “, 2017), EQUATE (Ravichander et al., 2019), DNC (Poliak et al.”
The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions (2020, methodology): “For NLI, we use SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) as the main standard datasets and use HANS (McCoy et al., 2019b), SNLI-hard (Gururangan et al., 2018), BREAK-NLI (Glockner et al., 2018), Stress Test (Naik et al., 2018a), SICK (Marelli et al., 2014), EQUATE (Ravichander et ”