QuestionBank: Creating a Corpus of Parse-Annotated Questions

By John Judge, Aoife Cahill, Josef van Genabith. 2006

117 citations recorded by the source corpus.

This paper describes the development of QuestionBank, a corpus of 4000 parse-annotated questions for (i) use in training parsers employed in QA, and (ii) evaluation of question parsing. We present a series of experiments to investigate the effectiveness of QuestionBank as both an exclusive and supplementary training resource for a state-of-the-art parser in parsing both question and non-question test sets. We introduce a new method for recovering empty nodes and their antecedents (capturing long distance dependencies) from parser output in CFG trees using LFG f-structure reentrancies. Our main findings are (i) using QuestionBank training data improves parser performance to 89.75% labelled bracketing f-score, an increase of almost 11% over the baseline; (ii) back-testing experiments on non-question data (Penn-II WSJ Section 23) shows that the retrained parser does not suffer a performance drop on non-question material; (iii) ablation experiments show that the size of training material provided by QuestionBank is sufficient to achieve optimal results; (iv) our method for recovering empty nodes captures long distance dependencies in questions from the ATIS corpus with high precision (96.82%) and low recall (39.38%). In summary, QuestionBank provides a useful new resource in parser-based QA research.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Grammar as a Foreign Language (2014, methodology): “QTB 1000 held-out sentences from the Question Treebank [9]; WEB the first half of each domain from the English Web Treebank [8] (8310 sentences).”
Learning Dependency-Based Compositional Semantics (2011, positive): “, 1993) and the Question Treebank (Judge et al., 2006)—thanks to Slav Petrov for providing the trained grammar.”
Globally Normalized Transition-Based Neural Networks (2016, methodology): “We conducted experiments on a number of different datasets: (1) the English Wall Street Journal (WSJ) part of the Penn Treebank (Marcus et al., 1993) with standard POS tagging splits; (2) the English “Treebank Union” multi-domain corpus containing data from the OntoNotes corpus version 5 (Hovy et al”
Structured Training for Neural Network Transition-Based Parsing (2015, methodology): “, 2006), the English Web Treebank (Petrov and McDonald, 2012), and the updated and corrected Question Treebank (Judge et al., 2006).”
Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples (2018, methodology): “If we try to run RSP on a more syntactically divergent corpus like QuestionBank6 (Judge et al., 2006), we find much more performance degradation.”
Syntactic Annotations for the Google Books NGram Corpus (2012, methodology): “First, we add the the Brown corpus and QuestionBank to the training data.”
From symbolic to sub-symbolic information in question classification (2011, methodology): “We use the Berkeley Parser (Petrov and Klein 2007) for the purpose of parsing questions, trained on the QuestionBank (Judge et al. 2006), a treebank of 4,000 parse-annotated questions.”
Dependency-based Convolutional Neural Networks for Sentence Embedding (2015, methodology): “8 but their parser is trained only on the QuestionBank (Judge et al., 2006) while we used the standard Stanford parser trained on both the Penn”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “…a certain length, and are written by certain demographics, the bias toward news articles had an impact on the linguistic phenomena studied in NLP (Judge et al., 2006), led to under-representation of challenges with handling longer documents (Beltagy et al., 2021), and had impact on early papers…”