Performance Issues and Error Analysis in an Open-Domain Question Answering System
By Dan I. Moldovan, Marius Pasca, Sanda M. Harabagiu, Mihai Surdeanu. 2002
359 citations recorded by the source corpus.
This paper presents an in-depth analysis of a state-of-the-art Question Answering system. Several scenarios are examined: (1) the performance of each module in a serial baseline system, (2) the impact of feedbacks and the insertion of a logic prover, and (3) the impact of various retrieval strategies and lexical resources. The main conclusion is that the overall performance depends on the depth of natural language processing resources and the tools used for answer finding.
Subfields: Artificial Intelligence, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Jianfeng Gao
- Graham Neubig
- Eric P. Xing
- Yejin Choi
- Yoshua Bengio
In Question Answering & Reading Comprehension
- Ming-Wei Chang
- William W. Cohen
- Wen-tau Yih
- Peter Clark
- Manzil Zaheer
- Graham Neubig
- Matthew Richardson
- Eric P. Xing
Citations in the corpus
- Learning Answer-Entailing Structures for Machine Comprehension (2015, background): “Mathematically this can be written as min w P and ∆ is the loss function between the predicted and the actual ranking and latent structure.”
- Differentiable Reasoning over a Virtual Knowledge Base (2020, background): “Advances in open-domain QA (Moldovan et al., 2002; Yang et al., 2019) suggest an alternative— instead of performing relation extraction, one could treat a large corpus as a virtual KB by answering queries with spans from the corpus.”
- Adaptive Document Retrieval for Deep Question Answering (2018, background): “Extensive research has analyzed the interplay of both document retrieval and machine comprehension in the context of knowledge-based systems (c. f. Moldovan et al., 2003) and even retrieval-based systems with machine learning (c. f. Brill et al., 2002).”
- Learning What is Essential in Questions (2017, neutral): “There is related work on extracting question type information (Li and Roth, 2002; Li et al., 2007) and applying it to the design and analysis of end-to-end QA systems (Moldovan et al., 2003).”
- Open domain question answering using Wikipedia-based knowledge model (2014, background): “Since multiple components involved in answering process, an incorrect answer should be traced back to identify the first place where the error occurred (Moldovan et al., 2003).”
- From symbolic to sub-symbolic information in question classification (2011, background): “Hence, in order to obtain good performance, it is of crucial importance for any question answering system to have an accurate question classifier [for a detailed analysis on the impact of question classification in question answering see Moldovan et al. (2003)].”
- Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, methodology): “hence important to the final output of the system [46].”
- Textual analysis of stock market prediction using breaking financial news: The AZFin text system (2009, methodology): “Building upon the Bag of Words approach, another tactic is to use a subset of terms as features [19], which can address issues related to article scaling while still encompassing the important concepts of an article [27].”
- A survey on question answering technology from an information retrieval perspective (2011, background): “, person name, date) [84].”
- Learning question classifiers: the role of semantic information (2005, background): “Recent works (Hovy, Gerber, Hermjakob, Lin and Ravichandran 2001; Moldovan, Pasca, Harabagiu and Surdeanu 2002; Roth, Cumby, Li, Morie, Nagarajan, Rizzolo, Small and Yih 2002) have shown that locating an accurate answer hinges on first filtering out a wide range of candidates based on some…”
- Information Retrieval: A Biomedical and Health Perspective (2020, background): “deemed the continued performing and reporting of underpowered clinical trials as an ethical dilemma [242].”
- Answer Extraction as Sequence Tagging with Tree Edit Distance (2013, methodology): “The features based on TED allow us to draw the connection between the question and answer sentences before answer extraction, whereas traditionally the exercise of answer validation (Magnini et al., 2002; Penas et al., 2008; Rodrigo et al., 2009) has been performed after as a remedy to ensure the…”
- Quantitative evaluation of passage retrieval algorithms for question answering (2003, background): “, [8, 16]) employ simple boolean queries for this reason, and because a boolean model allows for finer-grained control over query expansion.”
- Question Answering Using Enhanced Lexical Semantic Models (2013, background): “State-of-the-art QA systems often implement a complicated pipeline architecture, consisting of question analysis, document or passage retrieval, answer selection and verification (Ferrucci, 2012; Moldovan et al., 2003).”
- What Makes a Good Answer? The Role of Context in Question Answering (2003, background): “Although question answering systems have been studied extensively (Katz, 1997; Brill et al., 2001; Hovy et al., 2002; Katz et al., 2002; Moldovan et al., 2002; Lin et al., 2002), most research has focused on the underlying answer extraction algorithms.”