On Making Reading Comprehension More Comprehensive

By Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, Sewon Min. 2019

27 citations recorded by the source corpus.

Machine reading comprehension, the task of evaluating a machine’s ability to comprehend a passage of text, has seen a surge in popularity in recent years. There are many datasets that are targeted at reading comprehension, and many systems that perform as well as humans on some of these datasets. Despite all of this interest, there is no work that systematically defines what reading comprehension is. In this work, we justify a question answering approach to reading comprehension and describe the various kinds of questions one might use to more fully test a system’s comprehension of a passage, moving beyond questions that only probe local predicate-argument structures. The main pitfall of this approach is that questions can easily have surface cues or other biases that allow a model to shortcut the intended reasoning process. We discuss ways proposed in current literature to mitigate these shortcuts, and we conclude with recommendations for future dataset collection efforts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Question Answering & Reading Comprehension

Citations in the corpus

oLMpics-On What Language Model Pre-training Captures (2019, methodology): “, 2019b) and facilitates probing question involving complex and commonsense reasoning (Talmor et al., 2019; Gardner et al., 2019a; Talmor and Berant, 2018).”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, methodology): “In light of this, recent work has proposed approaches to pre-empting spurious correlations by designing annotation frameworks that better leverage human ‘‘common sense’’(40) and more critical approaches to dataset creation and use for tasks such as reading comprehension.(41)”
Learning from Task Descriptions (2020, methodology): “As we collect applicable inputs x for a task using a retrieval mechanism given the task description (Section 4), this also adds some lexical distance between the input and the description, to avoid simple lexical shortcuts to solving the task (Gardner et al., 2019).”
Evaluating Machines by their Real-World Language Use (2020, background): “Others in the community have observed similar issues, raising concerns about what today’s benchmark datasets measure (Yogatama et al., 2019; Kryscinski et al., 2019; McClelland et al., 2019; Gardner et al., 2019).”
Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge (2020, background): “ and somewhat unpredictable way [25, 26]. It similarly distinguishes our work from commonsense QA [27, 28, 29] and multi-hop QA, e.g., [30, 31, 32, 33, 34], where again model behavior is often opaque [35, 36, 37]. There have been numerous earlier demonstrations that neural systems can learn systemat”
Handling Bias in Toxic Speech Detection: A Survey (2022, methodology): “Consequently, the mitigation techniques in NLP have been helpful in tasks such as textual entailment [100], or reading comprehension [35].”
TuringAdvice: A Generative and Dynamic Evaluation of Language Use (2021, background): “Others in the community have observed similar issues, raising concerns about what today’s benchmark datasets measure (Yogatama et al., 2019; Kryscinski et al., 2019; McClelland 1 et al., 2019; Gardner et al., 2019).”
Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge (2020, background): “, [30, 31, 32, 33, 34], where again model behavior is often opaque [35, 36, 37].”