QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions

By Oyvind Tafjord, Matt Gardner, Kevin Lin, Peter Clark. 2019

123 citations recorded by the source corpus.

We introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships. QuaRTz contains general qualitative statements, e.g., “A sunscreen with a higher SPF protects the skin longer.”, twinned with 3864 crowdsourced situated questions, e.g., “Billy is wearing sunscreen with a lower SPF than Lucy. Who will be best protected from the sun?”, plus annotations of the properties being compared. Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships, and tests a system’s ability to comprehend and apply textual qualitative knowledge in a novel setting. We find state-of-the-art results are substantially (20%) below human performance, presenting an open challenge to the NLP community.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, neutral): “…et al., 2018), qasc (Khot et al., 2020), commonsense_qa (Talmor et al., 2019), ai2_arc (Clark et al., 2018), codah (Chen et al., 2019), superglue-copa (Gordon et al., 2012), dream (Sun et al., 2019), quartz-with_knowledge (Tafjord et al., 2019b), quartz-no_knowledge (Tafjord et al., 2019b).”
Transformers as Soft Reasoners over Language (2020, methodology): “While the non-synthetic nature of QuaRTz and ROPES adds realism to the task, it also complicates them as a test of “ma-chine reasoning”, in the sense of (producing the i/o behavior of) chaining rules to reach a conclusion.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…qa/multiple-choice qa Tafjord et al. 2019a quartz-no_knowledge qa/multiple-choice qa Tafjord et al. 2019b quartz-with_knowledge qa/multiple-choice qa Tafjord et al. 2019b quoref qa/machine reading comprehension Dasigi et al. 2019 race-high qa/multiple-choice qa Lai et al. 2017 race-middle…”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “Selsam et al. (2019), Tafjord et al. (2019), Lin et al. (2019), and Khashabi et al. (2020) designed their data collection process to include contrastive examples.”
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project (2019, background): “Further details and a detailed separate evaluation is available in (Tafjord et al., 2019).”
Competency Problems: On Finding and Removing Artifacts in Language Data (2021, background): “This suggests that care must be taken when performing edit-based data augmentation, as large edited training datasets are not likely to be artifact-free (cf. Tafjord et al., 2019; Huang et al., 2020).”
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022, methodology): “QuaRTz Sharon is conducting an experiment on valence electrons and soon discovers that when they are closer to the nucleus, they are _____ easily removed from the atom.”
REV: Information-Theoretic Evaluation of Free-Text Rationales (2022, background): “QuaRTz is an open-domain reasoning task about textual qualitative relationships.”
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning (2021, methodology): “The task of rule reasoning (Clark et al., 2020) is related to other recently proposed tasks on QA (We-ston et al., 2015; Yang et al., 2018; Lin et al., 2019; Tafjord et al., 2019; Richardson et al., 2020) and NLI (MacCartney and Manning, 2014).”
TIES-Merging: Resolving Interference When Merging Models (2023, methodology): “We finetune both T5-base and T5-large on seven tasks: question answering (QASC [33], WikiQA [87], and QuaRTz [73]), Paraphrase Identification (PAWS [90]), Sentence Completion (Story Cloze [68]), and Coreference Resolution (Winogrande [64] and WSC [37]).”