Crowdsourcing Multiple Choice Science Questions

By Johannes Welbl, Nelson F. Liu, Matt Gardner. 2017

891 citations recorded by the source corpus.

We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-specific text and a small set of existing questions. It produces model suggestions for document selection and answer distractor choice which aid the human question generation process. With this method we have assembled SciQ, a dataset of 13.7K multiple choice science exam questions. We demonstrate that the method produces in-domain questions by providing an analysis of this new dataset and by showing that humans cannot distinguish the crowdsourced questions from original questions. When using SciQ as additional training data to existing questions, we observe accuracy improvements on real science exams.

Subfields: Artificial Intelligence, Computer Science Applications, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “Morerecently, Welbl et al. (2017b) created the WikiHop dataset, containing questions that appear to require more than one Wikipedia document to answer (“multihop ques-tions”).”
Improving Question Answering with External Knowledge (2019, background): “In this paper, we focus on multiple-choice subject-area QA tasks, where the in-domain reference corpus does not provide sufficient relevant content on its own to answer a significant portion of the questions (Clark et al., 2016; Kobayashi et al., 2017; Welbl et al., 2017; Clark et al., 2018; Mihaylo”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “, 2013) (2013), SciQ (Welbl et al., 2017), TriviaQA (Joshi et al.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Tasks Type Random small core main ext HellaSwag (Zellers et al., 2019) Sentence completion 25.0 ✓ ✓ ✓ ✓ LAMBADA (Paperno et al., 2016) Sentence completion 0.0 ✓ ✓ ✓ Winogrande (Sakaguchi et al., 2021) Coreference resolution 50.0 ✓ ✓ ✓ ✓ PIQA (Bisk et al., 2020) Multiple-choice question answering 50.”
Hypothesis Only Baselines in Natural Language Inference (2018, methodology): “Hypotheses are assertions converted from question-answer pairs found in SciQ (Welbl et al., 2017).”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, background): “…2015b; Rajpurkar et al., 2016; Hewlett et al., 2016), web search queries (Nguyen et al., 2016), news articles (Hermann et al., 2015; Onishi et al., 2016), books (Hill et al., 2015; Paperno et al., 2016), science exams (Welbl et al., 2017), and trivia (Boyd-Graber et al., 2012; Dunn et al., 2017).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…cls/sentiment analysis Pang and Lee 2005 samsum cg/summarization Gliwa et al. 2019 scicite cls/other Cohan et al. 2019 sciq qa/multiple-choice qa Welbl et al. 2017 scitail cls/nli Khot et al. 2018 search_qa qa/closed-book qa Dunn et al. 2017 sick cls/nli Marelli et al. 2014 sms_spam cls/other…”
SciTaiL: A Textual Entailment Dataset from Science Question Answering (2018, methodology): “We use multiple-choice science questions from publicly released 4th grade (204 questions) and 8th grade (195 questions) exams4 and the crowd-sourced questions from SciQ dataset (2,835 questions) (Welbl, Liu, and Gardner 2017) to create Q.”
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020, background): “SciTail SciTail (Khot et al., 2018) is a textual entailment task built from multiple-choice science questions from 4th grade and 8th grade exams, as well as crowdsourced questions (Welbl et al., 2017).”
A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset (2018, background): “The SciQ Dataset (Welbl et al., 2017) contains 13,679 crowdsourced multiple choice science questions.”
Explanations for CommonsenseQA: New Dataset and Models (2021, background): “…either (i) proposing a new kind of dataset (Unger et al., 2014; Rajpurkar et al., 2016; Ling et al., 2017; Joshi et al., 2017; Trivedi et al., 2017; Welbl et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019; Talmor et al., 2019; Miao et al., 2020), or (ii) proposing a model with improved…”
CliCR: a Dataset of Clinical Case Reports for Machine Reading Comprehension (2018, methodology): “…exist primarily for the general domain; for specialized domains where background knowledge is crucial, crowdsourcing is intuitively less suitable (Welbl et al., 2017b), although some positive precedent exists for example in crowdsourcing annotations of radiology reports (Cocos et al., 2015).”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, positive): “To explore this, we evaluated some continual pre-trained models on M ATH P ILE (in Table 4) in several representative benchmarks including PIQA [8], ARC-Easy [15], ARC-Challenge [15], SciQ [64] and HellaSwag [69].”
A Systematic Review of Automatic Question Generation for Educational Purposes (2019, background): “– SciQ15 (Welbl et al. 2017) is a corpus of 13.7K science MCQs on biology, chemistry, earth science, and physics.”
Scaling Data-Constrained Language Models (2023, background): “[117] Johannes Welbl, Nelson F Liu, and Matt Gardner.”