ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension

By Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, Benjamin Van Durme. 2018

354 citations recorded by the source corpus.

We present a large-scale dataset, ReCoRD, for machine reading comprehension requiring commonsense reasoning. Experiments on this dataset demonstrate that the performance of state-of-the-art MRC systems fall far behind human performance. ReCoRD represents a challenge for future research to bridge the gap between human and machine commonsense reading comprehension. ReCoRD is available at this http URL

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Interestingly, on the reading comprehension tasks (MultiRC and ReCoRD) we exceed human performance by a large margin, suggesting the evaluation metrics used for these tasks may be biased towards machine-made predictions.”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “• ReCoRD is a reading comprehension dataset using paragraphs from CNN and Daily Mail news articles and automatically generated questions (Zhang et al., 2018).”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “swer (for example, a paragraph from an encyclopedia article) alongside the question (Rajpurkar et al., 2016; Zhang et al., 2018; Khashabi et al., 2018; Clark et al., 2019).”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “ReCoRD (Reading Comprehension with Commonsense Reasoning Dataset, Zhang et al., 2018) is a multiple-choice QA task.”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “…has seen a great deal of recent work (Ra-jpurkar et al., 2016; Joshi et al., 2017), and there have been many recent attempts to construct QA datasets that require advanced reasoning abilities (Yang et al., 2018; Welbl et al., 2018; Mi-haylov et al., 2018; Zellers et al., 2018; Zhang et al., 2018).”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “a recent trend in creating datasets with adversarial baselines in the loop (Paperno et al., 2016; Minervini and Riedel, 2018; Zellers et al., 2018; Zhang et al., 2019; Zellers et al., 2019).”
A Hybrid Neural Network Model for Commonsense Reasoning (2019, background): “, 2018), ReCoRD (Zhang et al., 2018), and so on, WSC and PDP better approximate real human reasoning, can be easily solved ar X iv :1 90 7.”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “Zero-shot: in addition to the 3 few-shot tasks, we evaluate on ReCoRD (Zhang et al., 2018), HellaSwag (Zellers et al.”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “ReCoRD (Zhang et al., 2018) • Struct to text tests the ability to describe some structured data using natural language.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, neutral): “…50.0 ✓ ✓ CB (De Marneffe et al., 2019) Natural language inference 33.3 ✓ ✓ RTE (Dagan et al., 2010) Natural language inference 50.0 ✓ ✓ ReCoRD (Zhang et al., 2018) Question answering 0.0 ✓ ANLI (Nie et al., 2019) Natural language inference 33.3 ✓ LogiQA (Liu et al., 2021) Multiple-choice…”
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, background): “‚ ReCoRD is a commonsense Question Answering dataset.”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, neutral): “Similarly, the ReCoRD dataset is a multiple choice dataset where the model must predict the masked out entity from a list of possible entities.”
Improving and Simplifying Pattern Exploiting Training (2021, methodology): “We use the same pre-trained model and hyperparameters as PET, except we increased the number of training batches to 1k and choose the best checkpoint on the dev set, since it has been shown that training longer can help even with few samples (Zhang et al., 2021).”
How many data points is a prompt worth? (2021, background): “We do not include ReCoRD (Zhang et al., 2018) in our comparisons as there is no head model to compare with, since it is already a cloze task.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “For ReCoRD (Zhang et al., 2018), given a passage p and a cloze question q, the task is to decide which”
The Effect of Natural Distribution Shift on Question Answering Models (2020, methodology): “Hendrycks et al. (2020) evaluates robustness to distribution shift for question answering, among other tasks, by carefully splitting subsets of the ReCoRD dataset (Zhang et al., 2018).”