Immediate-Head Parsing for Language Models

By Eugene Charniak. Annual Meeting of the Association for Computational Linguistics, 2001

362 citations recorded by the source corpus.

We present two language models based upon an "immediate-head" parser --- our name for a parser that conditions all events below a constituent c upon the head of c. While all of the most accurate statistical parsers are of the immediate-head variety, no previous grammatical language model uses this technology. The perplexity for both of these models significantly improve upon the trigram model base-line as well as the best previous grammar-based language model. For the better of our two models these improvements are 24% and 14% respectively. We also suggest that improvement of the underlying parser should significantly improve the model's perplexity and that even in the near term there is a lot of potential for improvement in immediate-head language models.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/436772d9a916f0382800cf18581cfdfd4f83c457

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Generating Sentences from Disentangled Syntactic and Semantic Spaces (2019, background): “Charniak (2001) and Clark (2001) pro-pose to utilize a top-down parsing mechanism for language modeling.”
The Importance of Syntactic Parsing and Inference in Semantic Role Labeling (2008, methodology): “To test this idea, we first build two SRL systems that use Collins’ parser (Collins, 1999) and Charniak’s parser (Charniak, 2001) respectively.”
Composition in Distributional Models of Semantics (2010, methodology): “Following previous work on syntactic language modeling (Roark, 2001; Charniak, 2001; Chelba and Jelinek, 1998), we trained the parser on sections 2–21 of the Penn Treebank containing 936,017 words. Note that Roark’s (2001) parser produces prefix probabilities for each word of a sentence which we con”
Calibrating Features for Semantic Role Labeling (2004, methodology): “They did not report results that use automatic parses with this version of the data, but using a previous version of the data, they reported an fscore of 79.4% using automatic parses (Charniak , 2001).”
Random forests and the data sparseness problem in language modeling (2007, methodology): “Following the success of the SLM as a language model, several different parsing-based language models have been proposed and studied (Charniak, 2001; Roark, 2001; Wang and Harper, 2002).”
Generative Incremental Dependency Parsing with Neural Networks (2015, methodology): “Syntactic language models based on PCFGs (Roark, 2001; Charniak, 2001) and incremental parsing (Chelba and Jelinek, 2000; Emami and Jelinek, 2005) have been proposed for speech recognition and machine translation.”
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (2018, methodology): “In models developed speciﬁcally for language modelling, there has been precedent for incorporating syntactic structure for the task (Roark, 2001; Charniak, 2001; Chelba & Jelinek, 2000).”
Exploiting prosodic breaks in language modeling with random forests (2008, background): “, [22, 23]) or tried to find the optimal back-off path or combination of paths [24, 25].”
Support Vector Learning for Semantic Argument Classification (2005, methodology): “To evaluate this scenario, we used the Charniak parser (Chaniak, 2001) to generate parses for PropBank training and test data. In these generated parses, there are about 6% arguments whose boundaries do not align exactly with any of the hand-generated phrase boundaries. We tried to recover training ”
Modelling and Optimizing on Syntactic N-Grams for Statistical Machine Translation (2015, methodology): “To a lesser extent, our work is similar to other parsing models that have been used for language modelling, such as lexicalized PCFGs (Charniak, 2001; Collins, 2003; Charniak et al., 2003), or structured language models (Chelba and Jelinek, 2000); previous efforts to include them in the translation ”
Identifying Real or Fake Articles: Towards better Language Modeling (2008, methodology): “There may be several reasons for this: (1) Our training data consisted of spoken transcripts from a broadcast news corpus whereas the Charniak Parser was trained on a different domain (Wall Street Journal) and (2) The parser was trained on mixed case text where as the data we used was all upper case”
Shallow Semantic Parsing using Support Vector Machines (2004, methodology): “This is calculated using a head word table described by (Magerman, 1994) and modified by (Collins, 1999, Appendix.”
Head-Driven Statistical Models for Natural Language Parsing (2003, background): “Charniak (2001) gives measurements of perplexity for a lexicalized PCFG. Gildea (2001) reports on experiments investigating the utility of different features in bigram lexical-dependency models for parsing.”
A bit of progress in language modeling (2001, methodology): “…advantage that when used in a stack-decoder, it allows sentence mixture models to be used with relatively little overhead, compared to Equation 8 Charniak (2001), as discussed in Section 10.5, uses a sentence level mixture model to combine a linguistic model with a trigram model, achieving…”
Accurate Unlexicalized Parsing (2003, background): “In the following decade, great success in terms of parse disambiguation and even language modeling was achieved by various lexicalized PCFG models (Magerman, 1995; Charniak, 1997; Collins, 1999; Charniak, 2000; Charniak, 2001).”
Tree Kernel-Based Relation Extraction with Context-Sensitive Structured Parse Tree Information (2007, methodology): “Both corpora are parsed using Charniak’s parser (Charniak, 2001) with the boundaries of all the entity mentions kept 4 .”