Deep Read: A Reading Comprehension System

By Lynette Hirschman, M. Light, Eric Breck, John D. Burger. 1999

This paper describes initial work on Deep Read, an automated reading comprehension system that accepts arbitrary text input (a story) and answers questions about it. We have acquired a corpus of 60 development and 60 test stories of 3rd to 6th grade material; each story is followed by short-answer questions (an answer key was also provided). We used these to construct and evaluate a baseline system that uses pattern matching (bag-of-words) techniques augmented with additional automated linguistic processing (stemming, name identification, semantic class identification, and pronoun resolution). This simple system retrieves the sentence containing the answer 30--40% of the time.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, methodology): “Remedia (Hirschman et al., 1999) also contains 5 different types of questions (based on question words) but is a much”
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text (2013, background): “The idea of using story-based reading comprehension questions to evaluate methods for machine reading itself goes back over a decade, when Hirschmann et al. (1999) showed that a bag of words approach, together with some heuristic linguistic modeling, could achieve 40% accuracy for the task of…”
Learning Question Classifiers (2002, background): “, 2001) and story comprehension (Hirschman et al., 1999) have become important directions in natural language processing.”
Modeling Biological Processes for Reading Comprehension (2014, background): “Hirschman et al. (1999) presented a bag-of-words approach to retrieving sentences for reading comprehension.”
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (2018, background): “While many RC datasets have been proposed over the years (Hirschman et al., 1999; Breck et al., 2001; Pe˜nas et al., 2011; Pe˜nas et al., 2012; Sutcliffe et al., 2013; Richardson et al., 2013; Berant et al., 2014), more recently, larger datasets have been proposed to accommodate the…”
What Makes Reading Comprehension Questions Easier? (2018, background): “One approach to doing so is the machine reading comprehension (MRC) task, in which a system answers questions about given texts (Hirschman et al., 1999).”
Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “Such examples are ubiquitous across natural language understanding (NLU) tasks such as reading comprehension (Hirschman et al., 1999) and recognizing textual entailment (Dagan et al., 2013), and Alice let Bob stand in front of her at the concert. even more so in tasks dedicated to commonsense…”
Relevance-guided Supervision for OpenQA with ColBERT (2020, background): “In recent work, the potential answers are usually selected from a set of pre-defined options (Hirschman et al., 1999; Richardson et al., 2013; Iyyer et al., 2014) or guaranteed to be a substring of the passage (Yang et al.”
Machine Comprehension with Syntax, Frames, and Semantics (2015, methodology): “Several comprehension tasks have been developed, including Remedia (Hirschman et al., 1999), CBC4kids (Breck et al., 2001), and the QA4MRE textual question answering tasks in the CLEF evaluations (Peñas et al., 2011; Peñas et al., 2013; Clark et al., 2012; Bhaskar et al., 2012).”
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets (2019, background): “Machine reading comprehension (MRC) is a testbed for evaluating natural language understanding (NLU), by letting machines answer questions about given texts (Hirschman et al. 1999).”
Multi-Mention Learning for Reading Comprehension with Neural Cascades (2017, methodology): “While data-driven approaches for the task date back to Hirschman et al. (1999), much of the recent progress can be attributed to new large-scale datasets such as the CNN/Daily Mail Corpus (Hermann et al., 2015), the Children’s Book Test Corpus (Hill et al., 2015) and the Stanford Question Answering…”
A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset (2018, methodology): “To explore reading comprehension as a research problem, Hirschman et al. (1999) manually created a dataset of 3rd and 6th grade reading comprehension questions with short answers.”
Solving logic puzzles: From robust processing to precise semantics (2004, background): “Moreover, the domain is another example of “ found test material” in the sense of (Hirschman et al., 1999): puzzle texts were developed with a goal independent of the evaluation of natural language processing systems, and so provide a more realistic evaluation framework than specially-designed…”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, background): “…work using question-answer pairs as a meaning representation (He et al., 2015; Michael et al., 2018), and more generally from the tradition of using question answering ability as a proxy for language understanding (Lehnert, 1977; Hirschman et al., 1999; Peñas et al., 2013; Richardson et al., 2013).”
Learning question classifiers: the role of semantic information (2005, background): “Open-domain question answering (Lehnert 1986; Harabagiu, Moldovan, Pasca, Mihalcea, Surdeanu, Bunescu, Girju, Rus and Morarescu 2000) and story comprehension (Hirschman et al. 1999) have become important directions in natural language processing.”
Computational Approaches to Sentence Completion (2012, background): “The DeepRead system (Hirschman et al., 1999) initiated a long line of research into reading comprehension based on test prep material (Charniak et al.”
Open-domain textual question answering techniques (2003, methodology): “Similar to Hirschman et al. (1999 any system working on Remedia questions.”