ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

By Weihao Yu, Zihang Jiang, Yanfei Dong, Jiashi Feng. 2020

343 citations recorded by the source corpus.

Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that the state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.

Subfields: Artificial Intelligence, Prompting & Reasoning, Evaluation & Benchmarks, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Percy Liang
  4. Yejin Choi
  5. Luke S. Zettlemoyer
  6. Yoav Goldberg
  7. Noah A. Smith
  8. Xiang Ren

In Prompting & Reasoning

  1. Xiang Ren
  2. Jian Yin
  3. Yining Hong
  4. Leyang Cui
  5. Pei Zhou
  6. Weizhu Chen
  7. Shizhuo Dylan Zhang
  8. Daya Guo

In Evaluation & Benchmarks

  1. Tongshuang Sherry Wu
  2. Yang Liu
  3. Ana Marasovic
  4. Duyu Tang
  5. Xiang Ren
  6. Yue Zhang
  7. Sarah Wiegreffe
  8. Weizhu Chen

In Question Answering & Reading Comprehension

  1. Xiang Ren
  2. Jun Zhao
  3. Song-Chun Zhu
  4. Pei Zhou
  5. Daniel S. Weld
  6. Weizhu Chen
  7. Kang Liu
  8. Jay Pujara

Citations in the corpus