Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

By Todor Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal. 2018

2,593 citations recorded by the source corpus.

We present a new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject. The open book that comes with our questions is a set of 1326 elementary level science facts. Roughly 6000 questions probe an understanding of these facts and their application to novel situations. This requires combining an open book fact (e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of armor is made of metal) obtained from other sources. While existing QA datasets over documents or knowledge bases, being generally self-contained, focus on linguistic understanding, OpenBookQA probes a deeper understanding of both the topic—in the context of common knowledge—and the language it is expressed in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art pre-trained QA methods perform surprisingly poorly, worse than several simple neural baselines we develop. Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts. We leave it as a challenge to solve the retrieval problem in this multi-hop setting and to close the large gap to human performance.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We consider eight standard common sense reasoning benchmarks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sak-aguchi et al., 2021), ARC easy and challenge (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018).”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We report the average of PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019a), WinoGrande (Sakaguchi et al., 2021), ARC easy and challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and CommonsenseQA (Talmor et al., 2018).”
Mistral 7B (2023, background): “…a wide variety of tasks categorized as follow: • Commonsense Reasoning (0-shot): Hellaswag [28], Winogrande [21], PIQA [4], SIQA [22], OpenbookQA [19], ARC-Easy, ARC-Challenge [9], CommonsenseQA [24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] • Reading Comprehension…”
Mixtral of Experts (2024, background): “…a wide variety of tasks categorized as follow: • Commonsense Reasoning (0-shot): Hellaswag [32], Winogrande [26], PIQA [3], SIQA [27], OpenbookQA [22], ARC-Easy, ARC-Challenge [8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] • Reading Comprehension…”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “The last, which we call transfer learning , leverages additional training examples from other multiple-choice QA tasks provided by the MMLU authors, namely MCTest (Richardson, 2013), RACE (Lai et al., 2017), ARC (Clark et al., 2018) and OBQA (Mihaylov et al., 2018) Additional benchmarks.”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, methodology): “While our definition of open-book is the same as in the OpenBookQA dataset introduced by Mihaylov et al. (2018), we do not directly address multi-hop inference in this work.”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “However, these attempts typically involve engineering data to be more difﬁcult by, for example, explicitly prompting users to write multi-step questions (Yang et al., 2018; Mihaylov et al., 2018), or ﬁltering out easy questions (Zellers et al., 2018).”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “, 2018), integrating knowledge from external sources (Mihaylov et al., 2018; Zhang et al., 2019), tracking entity state changes (Mishra et al.”
Improving Question Answering with External Knowledge (2019, methodology): “…dataset side, our work primarily focuses on multiple-choice examination datasets designed by educational experts (Lai et al., 2017; Clark et al., 2018; Mihaylov et al., 2018; Sun et al., 2019) since questions from these datasets are generally clean, error-free, and challenging (Sun et al., 2019).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “In contrast, although fine-tuning of MoE models produces substantial benefits for Storycloze, BoolQ, SST-2, MNLI and some improvements on OpenBookQA, it results in worse performance for HellaSwag, PIQA, and Winogrande.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, neutral): “…et al., 2020) Common Sense Reasoning: PIQA (Bisk et al., 2020), ARC (Easy) (Clark et al., 2018), ARC (Challenge) (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018) In-context Reading Comprehension: DROP (Dua et al., 2019), CoQA (Reddy et al., 2019), QuAC (Choi et al., 2018), SQuADv2…”
Measuring Massive Multitask Language Understanding (2020, background): “…either cover easy topics like grade school subjects for which models can already achieve strong performance (Clark et al., 2018; Khot et al., 2019; Mihaylov et al., 2018; Clark et al., 2019), or are focused on linguistic understanding in the form of reading comprehension (Lai et al., 2017;…”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “Reading comprehension datasets: BoolQ, MultiRC, OBQA.”
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, background): “Most multi-hop reasoning datasets do not fully annotate question decomposition (Yang et al., 2018; Khot et al., 2020a; Mihaylov et al., 2018).”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, background): “Science questions often require common sense, and have recently received attention (Clark et al., 2018; Mihaylov et al., 2018; Ostermann et al., 2018); however, they also need specialized scientific knowledge.”
OPT: Open Pre-trained Transformer Language Models (2022, background): “We evaluate our model on 16 standard NLP tasks utilized in the literature: HellaSwag (Zellers et al., 2019), StoryCloze (Mostafazadeh et al., 2016), PIQA (Bisk et al., 2020), ARC Easy and Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), WinoGrad (Levesque et al., 2011), Wino-Grand”