A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset

By Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue, Pavan Kapanipathi, Nicholas Mattei and 3 more. 2018

94 citations recorded by the source corpus.

The recent work of Clark et al. (2018) introduces the AI2 Reasoning Challenge (ARC) and the associated ARC dataset that partitions open domain, complex science questions into easy and challenge sets. That paper includes an analysis of 100 questions with respect to the types of knowledge and reasoning required to answer them; however, it does not include clear definitions of these types, nor does it offer information about the quality of the labels. We propose a comprehensive set of definitions of knowledge and reasoning types necessary for answering the questions in the ARC dataset. Using ten annotators and a sophisticated annotation interface, we analyze the distribution of labels across the challenge set and statistics related to them. Additionally, we demonstrate that although naive information retrieval methods return sentences that are irrelevant to answering the query, sufficient supporting text is often present in the (ARC) corpus. Evaluating with human-selected relevant sentences improves the performance of a neural machine comprehension model by 42 points.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Knowledge Graphs & Representation

Citations in the corpus

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, methodology): “While our focus is on language generation, we also evaluate the performance of quantized models on some popular zero-shot tasks, namely LAMBADA (Paperno et al., 2016), ARC (Easy and Challenge) (Boratko et al., 2018) and PIQA (Tata & Patel, 2003).”
QASC: A Dataset for Question Answering via Sentence Composition (2019, background): “However, these often require careful manual tuning due to the large variety of reasoning techniques needed for these questions (Boratko et al. 2018) and the large number of facts that often must be composed together (Jansen 2018; Jansen et al. 2016).”
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets (2019, methodology): “A skill-based analysis was conducted by Boratko et al. (2018), who proposed classiﬁcations of knowledge and reasoning.”
Improving Natural Language Inference Using External Knowledge in the Science Questions Domain (2018, methodology): “Our own previous work (Boratko et al. 2018) associates the QA task with the reasoning and knowledge types required to answer various types of science exam questions.”
ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning (2020, background): “…from the crowd-sourced development set, and marked each one with any number of the six categories which seemed necessary for the question as a simple approximation of prior works which examine the types of knowledge required for reasoning tasks (LoBue and Yates, 2011; Boratko et al., 2018).”
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge (2021, methodology): “, (Musa et al., 2019; Boratko et al., 2018; Ni et al., 2019; Xie et al., 2020), and as of January 2021 has 35 entries on its leaderboard2.”
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2019, background): “…challenging for QA (Clark et al., 2013; Clark, 2015; Clark et al., 2019), and existing science benchmarks are known to involve widespread use of such knowledge (see Clark et al. (2018); Boratko et al. (2018) for analysis), which is also arguably fundamental to more complex forms of reasoning.”
Answering Science Exam Questions Using Query Rewriting with Background Knowledge (2018, background): “As in this paper, Ni et al. (2018) pursue the approach suggested by Boratko et al. (2018) in learning how to transform a natural-language question into a query for which an IR system can return a higher-quality selection of results.”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “Different Row-Hessian Challenge.”