QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships

By Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, Ashish Sabharwal. 2018

85 citations recorded by the source corpus.

Many natural la guage questions require recognizing and reasoning with qualitative relationships (e.g., in science, economics, and medicine), but are challenging to answer with corpus-based methods. Qualitative modeling provides tools that support such reasoning, but the semantic parsing task of mapping questions into those models has formidable challenges. We present QUAREL, a dataset of diverse story questions involving qualitative relationships that characterize these challenges, and techniques that begin to address them. The dataset has 2771 questions relating 19 different types of quantities. For example, “Jenny observes that the robot vacuum cleaner moves slower on the living room carpet than on the bedroom carpet. Which carpet has more friction?” We contribute (1) a simple and flexible conceptual framework for representing these kinds of questions; (2) the QUAREL dataset, including logical forms, exemplifying the parsing challenges; and (3) two novel models for this task, built as extensions of type-constrained semantic parsing. The first of these models (called QUASP+) significantly outperforms off-the-shelf tools on QUAREL. The second (QUASP+ZERO) demonstrates zero-shot capability, i.e., the ability to handle new qualitative relationships without requiring additional training data, something not possible with previous models. This work thus makes inroads into answering complex, qualitative questions that require reasoning, and scaling to new relationships at low cost. The dataset and models are available at http://data.allenai.org/quarel.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Question Answering & Reading Comprehension

Citations in the corpus

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, neutral): “…et al., 2018), qasc (Khot et al., 2020), commonsense_qa (Talmor et al., 2019), ai2_arc (Clark et al., 2018), codah (Chen et al., 2019), superglue-copa (Gordon et al., 2012), dream (Sun et al., 2019), quartz-with_knowledge (Tafjord et al., 2019b), quartz-no_knowledge (Tafjord et al., 2019b).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…qa/multiple-choice qa Tafjord et al. 2019a quartz-no_knowledge qa/multiple-choice qa Tafjord et al. 2019b quartz-with_knowledge qa/multiple-choice qa Tafjord et al. 2019b quoref qa/machine reading comprehension Dasigi et al. 2019 race-high qa/multiple-choice qa Lai et al. 2017 race-middle…”
QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions (2019, background): “However, although QuaRel contains 2700 qualitative questions, its underlying qualitative knowledge was speciﬁed formally, using a small, ﬁxed ontology of 19 properties.”
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering (2020, methodology): “We terminate the training when we do not observe any performance improvements on the development set for 5 epochs for WIQA and 10 epochs for QuaRel, respectively.”
Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question Answering (2020, background): “…in recent years for natural language question answering such as (Rajpurkar et al., 2016; Joshi et al., 2017; Khashabi et al., 2018a; Richardson et al., 2013; Lai et al., 2017; Reddy et al., 2018; Choi et al., 2018; Tafjord et al., 2018; Mitra et al., 2019b; Sap et al., 2019; Bisk et al., 2019).”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “For example, DROP (Dua et al., 2019) is a MathQA dataset that requires discrete reasoning to answer questions such as “Which kicker kicked the most ﬁeld goals?” over the content of paragraphs.”
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022, methodology): “We collect such quantitative comparison questions by using the QuaRel dataset (Tafjord et al., 2019) containing questions from diverse ﬁelds such as physics and economics as the starting point.”
Careful Selection of Knowledge to Solve Open Book Question Answering (2019, background): “…et al., 2016; Joshi et al., 2017; Khashabi et al., 2018; Richardson et al., 2013; Lai et al., 2017; Reddy et al., 2018; Choi et al., 2018; Tafjord et al., 2018; Mitra et al., 2019) and many attempts have been made to solve these challenges (Devlin et al., 2018; Vaswani et al., 2017; Seo…”
Representing Numbers in NLP: a Survey and a Vision (2021, methodology): “(2020) modify the QuaRel dataset (Tafjord et al., 2019) to force models to perform quantity entailment, e.”
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks (2020, background): “As illustrated in Figure 6, most question sentences in Quarrel [Tafjord et al. , 2019] have high similarity implying that data is repetitive.”