Probabilistic Top-Down Parsing and Language Modeling

By Brian Roark. 2001

389 citations recorded by the source corpus.

This paper describes the functioning of a broad-coverage probabilistic top-down parser, and its application to the problem of language modeling for speech recognition. The paper first introduces key notions in language modeling and probabilistic parsing, and briefly reviews some previous approaches to using syntactic structure for language modeling. A lexicalized probabilistic top-down parser is then presented, which performs very well, in terms of both the accuracy of returned parses and the efficiency with which they are found, relative to the best broad-coverage statistical parsers. A new language model that utilizes probabilistic top-down parsing is then outlined, and empirical results show that it improves upon previous work in test corpus perplexity. Interpolation with a trigram model yields an exceptional improvement relative to the improvement observed by other models, demonstrating the degree to which the information captured by our parsing model is orthogonal to that captured by a trigram model. A small recognition experiment also demonstrates the utility of the model.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Recurrent Neural Network Grammars (2016, background): “It is likewise closely related to the “linearized” parse trees proposed by Vinyals et al. (2015) and to the top-down, left-to-right decompositions of trees used in previous generative parsing and language modeling work (Roark, 2001, 2004; Charniak, 2010).”
Targeted Syntactic Evaluation of Language Models (2018, background): “Syntax in LMs: There have been several proposals over the years to incorporate explicit syntax into LMs to overcome the inability of n-gram LMs to model long-distance dependencies (Jurafsky et al., 1995; Roark, 2001; Pauls and Klein, 2012).”
Head-Driven Statistical Models for Natural Language Parsing (2003, background): “Of particular relevance is other work on parsing the Penn WSJ Treebank (Jelinek et al. 1994; Magerman 1995; Eisner 1996a, 1996b; Collins 1996; Charniak 1997; Goodman 1997; Ratnaparkhi 1997; Chelba and Jelinek 1998; Roark 2001).”
A Bayesian Model for Generative Transition-based Dependency Parsing (2015, methodology): “Lexicalised PCFGs applied to language modelling (Roark, 2001; Charniak, 2001) show improvements over n-gram models, but decoding is prohibitively expensive for practical integration in language generation applications.”
Insensitivity of the Human Sentence-Processing System to Hierarchical Structure (2011, methodology): “, word-by-word) parser (Roark, 2001) was used to obtain the desired probability estimates for P(wt|w1 .”
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (2018, methodology): “In models developed speciﬁcally for language modelling, there has been precedent for incorporating syntactic structure for the task (Roark, 2001; Charniak, 2001; Chelba & Jelinek, 2000).”
Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. (2008, positive): “The parser used was Roark’s (2001) incremental top-down parser.”
Reading Level Assessment Using Support Vector Machines and Statistical Language Models (2005, methodology): “Future work includes testing additional classi(cid:2)er features, e.g. parser likelihood scores and features obtained using a syntax-based language model such as Chelba and Jelinek (2000) or Roark (2001).”
Robust garden path parsing (2004, methodology): “One of the beneﬁts of our parsing approach has been the application of our probabilistic parser as a language model for statistical speech recognition (Roark 2001a).”
Generative Incremental Dependency Parsing with Neural Networks (2015, methodology): “Syntactic language models based on PCFGs (Roark, 2001; Charniak, 2001) and incremental parsing (Chelba and Jelinek, 2000; Emami and Jelinek, 2005) have been proposed for speech recognition and machine translation.”
Speakers optimize information density through syntactic reduction (2006, methodology): “As in Collins (2003) and Roark (2001), this type of directed generative process allows conditioning on arbitrary features of the incremental utterance.”
Expectation-based syntactic comprehension. (2008, background): “However, evidence from the probabilistic parsing literature (Roark, 2001; Henderson, 2004) suggests that in typical sentences, most of the probability mass is focused on a small number of highly-ranked analyses.”
Supervised and unsupervised PCFG adaptation to novel domains (2003, methodology): “For the empirical trials, we used a top-down, left-to-right (incremental) statistical beam-search parser (Roark, 2001a; Roark, 2003).”
Coarse-to-Fine n-Best Parsing and MaxEnt Discriminative Reranking (2005, methodology): “At the end one has a beam-width’s number of best parses (Roark, 2001).”
Modeling the effects of memory on human online sentence processing with particle filters (2008, background): “Previous work has addressed the issues of feature locality and resource constraints by adop ting a pruning approach, in which hard locality constraints on probabilistic dependence are aban doned and only high-probability candidate structures are maintained after each step of incremental pa rsing [6,”
Top-Down Nearly-Context-Sensitive Parsing (2010, methodology): “This is a 20% error reduction over the previous best single-parser result of 86.8%, achieved by Rork(Roark, 2001).”
Assessing Relative Sentence Complexity using an Incremental CCG Parser (2016, background): “…parsing has shown to be useful for many real-time applications such as language modeling for speech recognition (Chelba and Jelinek, 2000; Roark, 2001), modeling text reading time (Demberg and Keller, 2008), dialogue systems (Stoness et al., 2004) and machine translation (Schwartz et…”