Head-Driven Statistical Models for Natural Language Parsing
By Michael Collins. 2003
2,083 citations recorded by the source corpus.
This article describes three statistical models for natural language parsing. The models extend methods from probabilistic context-free grammars to lexicalized grammars, leading to approaches in which a parse tree is represented as the sequence of decisions corresponding to a head-centered, top-down derivation of the tree. Independence assumptions then lead to parameters that encode the X-bar schema, subcategorization, ordering of complements, placement of adjuncts, bigram lexical dependencies, wh-movement, and preferences for close attachment. All of these preferences are expressed by probabilities conditioned on lexical heads. The models are evaluated on the Penn Wall Street Journal Treebank, showing that their accuracy is competitive with other models in the literature. To gain a better understanding of the models, we also give results on different constituent types, as well as a breakdown of precision/recall results in recovering various types of dependencies. We analyze various characteristics of the models through experiments on parsing accuracy, by collecting frequencies of various structures in the treebank, and through linguistically motivated examples. Finally, we compare the models to others that have been applied to parsing the treebank, aiming to give some explanation of the difference in performance of the various models.
Subfields: Artificial Intelligence, Natural Language Processing, Information Extraction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Noah A. Smith
- Christopher D. Manning
- Yoshua Bengio
- Jianfeng Gao
- Yoav Goldberg
In Natural Language Processing
- Christopher D. Manning
- Luke S. Zettlemoyer
- Noah A. Smith
- Percy Liang
- Mohit Bansal
- Mirella Lapata
- Yoav Goldberg
- Graham Neubig
In Information Extraction
- Daniel Klein
- Dan Roth
- Christopher D. Manning
- Wen-tau Yih
- Jason Aaron Edward Weston
- Vasin Punyakanok
- Ronan Collobert
- Dipanjan Das
Citations in the corpus
- Natural Language Processing (Almost) from Scratch (2011, methodology): “, 2005), related to CHUNK tags, and built using POS tags (Charniak, 2000; Collins, 1999).”
- Accurate Unlexicalized Parsing (2003, methodology): “…basic deficiency is that many rule types have been seen only once (and therefore have their probabilities overestimated), and many rules which occur in test sentences will never have been seen in training (and therefore have their probabilities underestimated – see Collins (1999) for analysis).”
- Generating Typed Dependency Parses from Phrase Structure Parses (2006, background): “Recent years have seen the introduction of a number of treebank-trained statistical parsers [Collins (Collins, 1999), Charniak (Charniak, 2000), Stanford (Klein and Manning, 2003)] capable of generating parses with high accuracy.”
- Search-based structured prediction (2009, positive): “We begin by generating dependency parse trees of each sentence in the document collection (an example such tree is shown in Figure 2) using an in-house implementation of Model II ( Collins, 2003 ) and hand-written head rules.”
- A theory of learning from different domains (2010, background): “In general, labeled data for tasks like part-of-speech tagging (Ratnaparkhi 1996), parsing (Collins 1999), information extraction (Bikel et al. 1997), and sentiment analysis (Pang et al. 2002) are drawn from a limited set of document types and genres in a given language due to availability, cost,…”
- Modeling Annotators: A Generative Approach to Learning from Annotator Rationales (2008, methodology): “(14)We parse each sentence with the Collins parser (Collins, 1999).”
- Globally Normalized Transition-Based Neural Networks (2016, background): “However the label bias problem (see Bottou (1991), Collins (1999) pages 222-226, Lafferty et al. (2001), Bottou and LeCun (2005), Smith and Johnson (2007)) means that locally normalized models often have a very weak ability to revise earlier decisions.”
- End-to-end learning of semantic role labeling using recurrent neural networks (2015, methodology): “In the work of (Pradhan et al., 2005), the syntactic tags are produced by Charniak parser (Char-niak, 2000; Charniak and Johnson, 2005) and Collins parser (Collins, 2003) respectively.”
- Unbounded Dependency Recovery for Parser Evaluation (2009, background): “…score is an aggregate over a highly skewed distribution of all constituent types; evaluations which look at individual constituent or dependency types show that the accuracies on some, semantically important, constructions, such as coordination and PP-attachment, are much lower (Collins, 1999).”
- Sum-product networks: A new deep architecture (2011, background): “Probabilistic context-free grammars and statistical pars ing [6] can be straightforwardly implemented as decomposable SPNs, with non-terminal nodes corresponding to sums (or maxes) and productions corresponding to products (logical conjunctions for standard PCFGs, and general products for head-drive”
- Discriminative Training of a Neural Network Statistical Parser (2004, methodology): “…are computed with the evalb program following the standard criteria in (Collins, 1999), and using the standard training (sections 2{22, 39,832 sentences, 910,196 words), validation (section 24, 1346 sentence, 31507 words), and testing (section 23, 2416 sentences, 54268 words) sets (Collins, 1999).”
- Self-Training PCFG Grammars with Latent Annotations Across Languages (2009, background): “There is an extensive research literature on building high quality parsers for English (Collins, 1999; Charniak, 2000; Charniak and Johnson, 2005; Petrov et al., 2006), however, models for parsing other languages are less well developed.”
- The Proposition Bank: An Annotated Corpus of Semantic Roles (2005, neutral): “The system first passed sentences through an automatic parser (Collins 1999), extracted syntactic features from the parses, and estimated probabilities for semantic roles from the syntactic and lexical features.”
- Learning Accurate, Compact, and Interpretable Tree Annotation (2006, methodology): “Probabilistic context-free grammars (PCFGs) underlie most high-performance parsers in one way or another (Collins, 1999; Charniak, 2000; Charniak and Johnson, 2005).”
- Measuring the Evolution of a Scientific Field through Citation Frames (2018, background): “…Pereira and Warren, 1983; Haas, 1989), with such papers receiving at most three citations that year; whereas in 2000, most comparisons were to a core set of parsing papers, e.g., (Collins, 1999; Buchholz et al., 1999; Collins, 1997), with a much sharper (lower entropy) distribution of citations.”
- Analysing Data-To-Text Generation Benchmarks (2017, neutral): “Parses are obtained using Collins’ constituent parser (Collins, 1999).”
- Solving logic puzzles: From robust processing to precise semantics (2004, background): “Thus, state-of-the-art work on probabilistic parsing (e.g., (Collins, 1999)) provides a good solution to robust, broad coverage parsing with automatic and frequently successful ambiguity resolution, but has largely ignored issues of semantic interpretation.”
- MaltParser: A Data-Driven Parser-Generator for Dependency Parsing (2006, background): “History-based feature models for predicting the next parser action (Black et al., 1992; Magerman, 1995; Ratnaparkhi, 1997; Collins, 1999)”