A Hierarchical Bayesian Language Model Based On Pitman-Yor Processes

By Yee Whye Teh. 2006

613 citations recorded by the source corpus.

We propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which produce power-law distributions more closely resembling those in natural languages. We show that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods for n-gram language models. Experiments verify that our model gives cross entropy results superior to interpolated Kneser-Ney and comparable to modified Kneser-Ney.

Subfields: Artificial Intelligence, Natural Language Processing, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Probabilistic & Bayesian Methods

Citations in the corpus

Experience Grounds Language (2020, background): “…et al., 2012) made much of WS2 advances possible, but perception, interaction, and social robotics bring new challenges which our current tools are not equipped to handle. with Good (1953), the weights of Kneser and Ney (1995); Chen and Goodman (1996), and the power-law distributions of Teh (2006).”
Dirichlet Process (2017, result): “This construction is also significantly more straightforward and general than previous proofs of the existence of DPs.”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “…(1995) noticed that a Bayesian view of autoregressive language models may prove beneﬁcial, reinterpreting smoothing and backoff in 𝑛 -gram models as inference in a hierarchical model where higher-order distributions are drawn from a Dirichlet distribution whose mean is a lower-order distributions.”
Mining idioms from source code (2014, methodology): “Source code has been shown to be highly repetitive [14], suggesting that statistical NLP methods could be promising for code analysis.”
Hierarchical Bayesian Language Models for Conversational Speech Recognition (2010, background): “Approaches which aim to improve on maximum-likelihood -gram models of word sequences include neural network-based models [8], latent variable models [9], and a Bayesian framework [10]–[12].”
Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model (2018, methodology): “ types. 7Baayen and Sproat (1996) argue for using only the hapax legomena (words that only appear once) to predict the behavior of rare and unknown words. The Bayesian approach (MacKay and Peto 1995; Teh 2006) is a compromise: frequent word types are also used, but they have no more inﬂuence than in”
A General Method for Robust Bayesian Modeling (2015, background): “Modern Bayesian modeling has had an impact on many ﬁelds, including natural language processing (Blei et al., 2003; Teh, 2006), computer vision (Fei-Fei and Perona, 2005), the natural sciences (Pritchard et al., 2000), and the social sciences (Grimmer, 2009).”
A Structural Smoothing Framework For Robust Graph Comparison (2015, methodology): “A novel interpretation of interpolated Kneser-Ney is proposed by [25] as approximate inference in a hierarchical Bayesian model consisting of Pitman-Yor process [16].”
A Bayesian Model for Generative Transition-based Dependency Parsing (2015, methodology): “HPYP models were originally proposed for n-gram language modelling (Teh, 2006), and have been applied to various NLP tasks.”
Shared Segmentation of Natural Scenes Using Dependent Pitman-Yor Processes (2008, methodology): “N-gram language models based on HPY processes [15, 16] have somewhat different forms.”