Adaptor Grammars: A Framework for Specifying Compositional Nonparametric Bayesian Models

By Mark Johnson, T. Griffiths, Sharon Goldwater. 2006

332 citations recorded by the source corpus.

This paper introduces adaptor grammars, a class of probabilistic models of language that generalize probabilistic context-free grammars (PCFGs). Adaptor grammars augment the probabilistic rules of PCFGs with "adaptors" that can induce dependencies among successive uses. With a particular choice of adaptor, based on the Pitman-Yor process, nonparametric Bayesian models of language using Dirichlet processes and hierarchical Dirichlet processes can be written as simple grammars. We present a general-purpose inference algorithm for adaptor grammars, making it easy to define and use such models, and illustrate how several existing nonparametric Bayesian models can be expressed within this framework.

Subfields: Artificial Intelligence, Natural Language Processing, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Probabilistic & Bayesian Methods

Citations in the corpus

An Introduction to Probabilistic Programming (2018, background): “adaptor grammars (Johnson et al., 2007)), they are at best cumbersome to denote mathematically.”
Grammar Variational Autoencoder (2017, background): “Modern approaches allow the probabilities used to at each stage to depend on the current state of the parse tree (Johnson et al., 2007).”
Church: a language for generative models (2008, methodology): “If we memoize unfold and use this adapted-unfold on PCFG transitions we recover the Adaptor Grammar model of [5]; if we similarly “adapt” the HDP-PCFG or HDP-HMM we get interesting new models that have not been considered in the literature.”
The Zero Resource Speech Challenge 2015: Proposed Approaches and Results (2016, methodology): “2 For the topline we evaluated the patterns discovered by a an adaptor grammar [19] system based on the phoneme annotation.”
Venture: a higher-order probabilistic programming platform with programmable inference (2014, background): “Examples include probabilistic grammars (Jelinek et al., 1992), nonparametric Bayesian models (Rasmussen, 1999; Johnson et al., 2007; Rasmussen and Williams, 2006; Griffiths and Ghahramani, 2005), probabilistic models over worlds with a priori unknown numbers of objects, models for learning the…”
Automatic Sampler Discovery via Probabilistic Programming and Approximate Bayesian Computation (2016, background): “Refer to the probabilistic program code in Figure 1 where the first line establishes a correspondence between T and the variable program-text then samples it from p(T ) where grammar is an adaptor-grammar-like [14] prior on program text that is described in Section 3.”
Discovering linguistic structures in speech: models and applications (2014, methodology): “Another model that is worth noting are adaptor grammars [83], which have proven to be a powerful framework for word segmentation.”
Productivity and Reuse in Language (2011, unknown): “Full–Parsing: We formalize the full–parsing model using Multinomial–Dirichlet Probabilistic Context–Free Grammars (MDPCFG) (Johnson et al., 2007b).”
Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model (2018, unknown): “(2015) Kawakami et al. (2017), Hwang and Sung (2017), This work”
A Tutorial on Bayesian Nonparametric Models (2011, background): “, 2006), data in a tree (Johnson et al., 2007; Liang et al., 2007), relational data (Kemp et al.”
The zero resource speech challenge 2015 (2015, background): “The topline is a unigram Adaptor Grammar [21] run on the gold phoneme transcription.”
A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition (2013, methodology): “Finally, we used the ’Colloc’ model from [21], which is based on the Adaptor Grammar framework [28] and assumes that the latent word sequence is generated as a sequence of “collocations,”, each consisting of one or more words.”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “11As mentioned before, the idea of spelling out words in isolation from hidden states had previously proven unsuccessful in Jozefowicz et al. (2016)’s comparison, but this was in a closed-vocab setup and without the caching mechanism Kawakami et al. (2017) employ.”
Inducing Tree-Substitution Grammars (2010, methodology): “Adaptor Grammars (Johnson et al., 2007b) are another recent nonparametric Bayesian model for learning hierarchical structure from strings.”
Covariance in Unsupervised Learning of Probabilistic Grammars (2010, methodology): “Variational approximations (Johnson, 2007; Kurihara and Sato, 2006)and randomized sampling approximations (Johnson et al., 2006; Goldwater, 2006) are typically appied.”
A Machine Learning Framework for Programming by Example (2013, background): “It would also be interesting to incorporate ideas like adaptor grammars (Johnson et al., 2006) and learning program structure as in (Liang et al.”
Learning phonetic categories by learning a lexicon (2009, methodology): “Following previous work in morphology (Goldwater, Griffiths, & Johnson, 2006), word segmentation (Goldwater, Griffiths, & Johnson, in press), and grammar learning ( Johnson, Griffiths, & Goldwater, 2007 ), learners’ prior beliefs about the phoneme inventory are encoded using a nonparametric Bayesian”