RACE: Large-scale ReAding Comprehension Dataset From Examinations

By Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Eduard H. Hovy. 2017

1,655 citations recorded by the source corpus.

We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students’ ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at https://github.com/qizhex/RACE_AR_baselines.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, neutral): “Other recently released datasets include (Lai et al., 2017).”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “RACE has signiﬁcantly longer context than other popular reading comprehension datasets and the proportion of questions that requires reasoning is very large.”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, neutral): “The RACE dataset [17] contains near 100K questions taken from the English exams for middle and high school Chinese students in the age range between 12 to 18, with the answers generated by human experts.”
Training Compute-Optimal Large Language Models (2022, neutral): “• Reading comprehension on RACE (Lai et al., 2017) • Common sense understanding on HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), Wino-grande (Sakaguchi et al., 2020), SIQA (Sap et al., 2019), BoolQ (Clark et al., 2019), and TruthfulQA (Lin et al., 2021).”
PaLM: Scaling Language Modeling with Pathways (2022, background): “…In-context Reading Comprehension: DROP (Dua et al., 2019), CoQA (Reddy et al., 2018), QuAC (Choi et al., 2018), SQuADv2 (Rajpurkar et al., 2018), RACE (Lai et al., 2017) • SuperGLUE (Wang et al., 2019a) • Natural Language Inference (NLI): Adversarial NLI (Nie et al., 2019) Table 4 includes…”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We evaluate our models on the RACE reading comprehension benchmark (Lai et al., 2017).”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “The last, which we call transfer learning, leverages additional training examples from other multiple-choice QA tasks provided by the MMLU authors, namely MCTest (Richardson, 2013), RACE (Lai et al., 2017), ARC (Clark et al.”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “For the AB format, we use ROUGE-L metric (Lin et al., 2006; Min et al., 2019; Nishida et al., 2019).”
NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “In recent years, researchers have published a large number of annotated MRC datasets such as CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016), RACE (Lai et al., 2017), Trivi-aQA (Joshi et al., 2017) and so on.”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “One of the most compelling signs of these breakthroughs is the evolution of machine performance on a reading comprehension task designed for middle and high-school English exams in China, the RACE test (Lai et al., 2017): the paper that originally describes the task and formulates the modeling chall”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “, 2016; 2018), and the reading comprehension RACE dataset (Lai et al., 2017).”
Natural Questions: A Benchmark for Question Answering Research (2019, background): “There are a number of reading comprehension benchmarks based on multiple choice tests (Mihaylov et al., 2018; Richardson et al., 2013; Lai et al., 2017).”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “There are other large-scale datasets (Nguyen et al., 2016; Joshi et al., 2017; Lai et al., 2017; Dunn et al., 2017; Kocisky et al., 2018; Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018) targeting different aspects of reading comprehension.”
CoQA: A Conversational Question Answering Challenge (2018, background): “…children’s stories from MCTest (Richardson et al., 2013), literature from Project Gutenberg,5 middle and high school English exams from RACE (Lai et al., 2017), news articles from CNN (Hermann et al., 2015), articles from Wikipedia, Reddit articles from the Writing Prompts dataset (Fan et…”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, methodology): “Finally, some datasets, like MCTest (Richard-son et al., 2013) and RACE (Lai et al., 2017), pose multiple choice questions, which can have a “none of the above” option.”
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, methodology): “, 2015), suffer from even lower percentage of such questions (12% and 2% respectively (Lai et al., 2017)).”
Option Comparison Network for Multiple-choice Reading Comprehension (2019, methodology): “Therefore, RACE is extremely challenging for MCRC models.”
Improving Question Answering with External Knowledge (2019, methodology): “, 2019) on a large-scale multiple-choice machine reading comprehension dataset RACE (Lai et al., 2017) collected from English-as-a-foreign-language exams, which provides a ground truth reference document instead of a reference corpus for each question.”