Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
By Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord. 2018
5,112 citations recorded by the source corpus.
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.
Subfields: Artificial Intelligence, Prompting & Reasoning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Quoc V. Le
- Hannaneh Hajishirzi
- Graham Neubig
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Denny Zhou
- Hannaneh Hajishirzi
- Yejin Choi
- Peter Clark
- Swaroop Mishra
- Tushar Khot
- Ashish Sabharwal
In Evaluation & Benchmarks
- Mohit Bansal
- Hannaneh Hajishirzi
- Ashish Sabharwal
- Yejin Choi
- Luke S. Zettlemoyer
- Samuel R. Bowman
- Tushar Khot
- Chitta Baral
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, background): “The benchmarks include: • Open-Domain Closed-Book Question Answering tasks:5 TriviaQA (Joshi et al., 2017), Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) • Cloze and Completion tasks: LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), StoryCloze (Mo”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We consider eight standard common sense reasoning benchmarks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sak-aguchi et al., 2021), ARC easy and challenge (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018).”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “, 2021), ARC easy and challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al.”
- Mistral 7B (2023, background): “…as follow: • Commonsense Reasoning (0-shot): Hellaswag [28], Winogrande [21], PIQA [4], SIQA [22], OpenbookQA [19], ARC-Easy, ARC-Challenge [9], CommonsenseQA [24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] • Reading Comprehension (0-shot): BoolQ [8], QuAC [7] • Math:…”
- Mixtral of Experts (2024, background): “…as follow: • Commonsense Reasoning (0-shot): Hellaswag [32], Winogrande [26], PIQA [3], SIQA [27], OpenbookQA [22], ARC-Easy, ARC-Challenge [8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] • Reading Comprehension (0-shot): BoolQ [7], QuAC [5] • Math:…”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “, 2017), ARC (Clark et al., 2018) and OBQA (Mihaylov et al.”
- UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “Specifically, MC questions without any context paragraph are encoded as question \n (A) c1 (B) c2 . . . where c1 , c1 , . . . are the set of candidate answers (see the example from ARC dataset).”
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “We use the adaptation of this model to multiple-choice question answering proposed by Clark et al. (2018), which works as follows: (1) convert q and each c i into a hypothesis, h i , and each retrieved fact into a premise p j ; and (2) return the answer choice with the highest entailment score, arg…”
- Improving Question Answering with External Knowledge (2019, methodology): “On the dataset side, our work primarily focuses on multiple-choice examination datasets designed by educational experts (Lai et al., 2017; Clark et al., 2018; Mihaylov et al., 2018; Sun et al., 2019) since questions from these datasets are generally clean, error-free, and challenging (Sun et al.,…”
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “, 2018), ARC (Challenge) (Clark et al., 2018), OpenBookQA (Mihaylov et al.”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “Numerous NLP benchmarks test models on factual questions [3, 7, 16, 41].”
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, background): “, 2016) and the ARC Reasoning Challenge (Clark et al., 2018).”
- Measuring Massive Multitask Language Understanding (2020, background): “Most either cover easy topics like grade school subjects for which models can already achieve strong performance (Clark et al., 2018; Khot et al., 2019; Mihaylov et al., 2018; Clark et al., 2019), or are focused on linguistic understanding in the form of reading comprehension (Lai et al.”
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “, 2021) for open-domain question-answering with implicit multi-hop reasoning, and the AI2 Reasoning Challenge (ARC) (Clark et al., 2018).”
- Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “(Clark et al., 2018), FLAN substantially outperforms both zero-shot and few-shot GPT-3.”
- CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, background): “Science questions often require common sense, and have recently received attention (Clark et al., 2018; Mi-haylov et al., 2018; Ostermann et al., 2018); however, they also need specialized scientific knowledge.”
- OPT: Open Pre-trained Transformer Language Models (2022, background): “Our performance roughly matched GPT-3 for 10 tasks, and underperformed in 3 tasks (ARC Challenge and MultiRC).”
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “Due to space constraints a representative subset of the results are shown here, with the rest in Appendix E. Natural Language Tasks We evaluate our model on a diverse collection of standard language model evaluation datasets: ANLI (Nie et al., 2020), ARC (Clark et al., 2018), HeadQA (English) (Vilar”
- The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, neutral): “…completion 0.0 ✓ ✓ ✓ Winogrande (Sakaguchi et al., 2021) Coreference resolution 50.0 ✓ ✓ ✓ ✓ PIQA (Bisk et al., 2020) Multiple-choice question answering 50.0 ✓ ✓ ✓ ✓ ARC (Clark et al., 2018) Natural language inference 25.0 ✓ ✓ ✓ ✓ OpenBookQA (Mihaylov et al., 2018) Multiple-choice question…”