ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
By Weihao Yu, Zihang Jiang, Yanfei Dong, Jiashi Feng. 2020
343 citations recorded by the source corpus.
Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that the state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.
Subfields: Artificial Intelligence, Prompting & Reasoning, Evaluation & Benchmarks, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Luke S. Zettlemoyer
- Yoav Goldberg
- Noah A. Smith
- Xiang Ren
In Prompting & Reasoning
In Evaluation & Benchmarks
- Tongshuang Sherry Wu
- Yang Liu
- Ana Marasovic
- Duyu Tang
- Xiang Ren
- Yue Zhang
- Sarah Wiegreffe
- Weizhu Chen
In Question Answering & Reading Comprehension
Citations in the corpus
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “Examples include TriviaQA (Joshi et al., 2017) which contains quiz league questions, ReClor (Yu et al., 2019) which contains logic puzzles from law school admissions tests, CommonsenseQA (Talmor et al., 2019), which targets relations in ConceptNet, and MMLU (Hendrycks et al., 2020), which…”
- Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text (2021, methodology): “Figure 1: A logical reasoning example from ReClor dataset (Yu et al., 2020).”
- Competition-Level Problems are Effective LLM Evaluators (2023, background): “…et al., 2021), numerical reasoning (Dua et al., 2019), multi-hop reasoning (Yang et al., 2018), arithmetic reasoning (Patel et al., 2021; Cobbe et al., 2021), structured reasoning (Yu et al., 2018; Lei et al., 2023), inductive reasoning (Sinha et al., 2019) and logical reasoning (Yu et al., 2020).”
- On the Paradox of Learning to Reason from Data (2022, background): “Yet real-world logical reasoning benchmarks (Patel et al., 2021; Welleck et al., 2021; Yu et al., 2020) are strictly more difficult than SimpleLogic as models also need to handle language variance and incorporate prior knowledge.”
- From LSAT: The Progress and Challenges of Complex Reasoning (2021, methodology): “It is a challenging task widely studied in recent years and several logical reasoning benchmarks have been introduced, such as ReClor [14] and LogiQA [25].”
- Large Language Models are Better Reasoners with Self-Verification (2022, background): “…2022), including arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2016; Patel et al., 2021; Cobbe et al., 2021), commonsense reasoning (Talmor et al., 2018; Bhagavatula et al., 2019; Geva et al., 2021; Zhu et al., 2022b), and logical reasoning (Liu et al., 2020; Yu et al., 2020).”
- On the Advance of Making Language Models Better Reasoners (2022, background): “…2018), arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive reasoning (Sinha et al., 2019) and tabular reasoning (Chen et al., 2020; Zhu et al., 2021).”
- Learning by Fixing: Solving Math Word Problems with Weak Supervision (2020, background): “Previous studies of MWPs range from traditional rule-based methods (Fletcher 1985; Bakman 2007; Yu-hui et al. 2010), statistical learning methods (Kushman et al. 2014; Zhou, Dai, and Chen 2015; Mitra and Baral 2016; Roy and Roth 2017; Huang et al. 2016), semantic-parsing meth-ods (Shi et al. 2015;…”
- Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, background): “…(Kaushik and Lipton, 2018) bAbI Partial Baselines (Kaushik and Lipton, 2018) SearchQA Manual Analyses (Pugaliya et al., 2019) ReClor Heuristics (Yu et al., 2020) CambridgeDialogs Arch/Training Improvements (Grail et al., 2018) QuAC Manual Analyses (Yatskar, 2019) The following table shows…”
- Making Language Models Better Reasoners with Step-Aware Verifier (2022, background): “…2018), arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive reasoning (Sinha et al., 2019) and tabular reasoning (Chen et al., 2020; Zhu et al., 2021).”
- Commonsense-Focused Dialogues for Response Generation: An Empirical Study (2021, background): “There have been many benchmarks that focus on reasoning abilities in multiple tasks such as reading comprehension (Huang et al., 2019; Yu et al., 2020), dialogue systems (Cui et al.”
- Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “(2021) collect a few high-quality explanations from LSAT law exam preparation books to accompany the ReClor dataset (Yu et al., 2020), but the authors have to manually edit and shorten the explanations.”
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, methodology): “We finetuned a RoBERTa model on ReClor data [69], which contains questions from standardized exams like the LSAT and GMAT.”
- Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts (2020, methodology): “2020) Next Utterance Prediction 3 Dialogue Exam ReClor (Yu et al. 2020) Reading Comprehension 3 Passage Exam LogiQA (Liu et al.”
- UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training (2022, unknown): “Small 79.4 80.7 67.6 66.4 77.1 78.0 45.5 53.7 55.9 55.2 50.4 50.0 42.9 47.1 35.9 37.0 80.0 72.8 59.4 60.1 0.7 60 Base 88.2 86.8 78.1 74.7 82.5 82.4 49.8 60.4 70.3 69.0 59.0 59.4 53.0 58.5 42.4 45.7 86.9 86.3 67.8 69.2 1.5 50 Large 91.1 90.8 85.9 86.3 86.1 85.8 51.6 63.2 78.0 78.8 68.2 71.8 65.9 70.5”