Looking Under the Hood: Tools for Diagnosing Your Question Answering Engine

By Eric Breck, M. Light, Gideon S. Mann, Ellen Riloff, B. Brown, Pranav Anand, Mats Rooth, Michael Thelen. 2001

30 citations recorded by the source corpus.

In this paper we analyze two question answering tasks: the TREC-8 question answering task and a set of reading comprehension exams. First, we show that Q/A systems perform better when there are multiple answer opportunities per question. Next, we analyze common approaches to two subproblems: term overlap for answer sentence identification, and answer typing for short answer extraction. We present general tools for analyzing the strengths and limitations of techniques for these sub-problems. Our results quantify the limitations of both term overlap and answer typing to distinguish between competing answer candidates.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text (2013, background): “Earlier work has also aimed at controlling the scope by limiting the text to children’s stories: Breck et al. (2001) collected 75 stories from the Canadian Broadcasting Corporation’s web site for children, and generated 650 questions for them manually, where each question was answered by a sentence…”
Machine Comprehension with Syntax, Frames, and Semantics (2015, methodology): “Several comprehension tasks have been developed, including Remedia (Hirschman et al., 1999), CBC4kids (Breck et al., 2001), and the QA4MRE textual question answering tasks in the CLEF evaluations (Peñas et al., 2011; Peñas et al., 2013; Clark et al., 2012; Bhaskar et al., 2012).”
Automatic Multi-Layer Corpus Annotation for Evaluation Question Answering Methods: CBC4Kids (2003, methodology): “Starting from MITRE’s CBC4Kids corpus collection (Breck et al., 2001), we have created a practical multi-layer annotation scheme and added various strata of linguistic annotation automatically 1 using state-of-the- 1i.e. there is no gold standard for the linguistic annotation, but task-based “gold…”