Dependent Multinomial Models Made Easy: Stick-Breaking with the Polya-gamma Augmentation

By Scott W. Linderman, Matthew J. Johnson, Ryan P. Adams. 2015

107 citations recorded by the source corpus.

Many practical modeling problems involve discrete data that are best represented as draws from multinomial or categorical distributions. For example, nucleotides in a DNA sequence, children's names in a given state and year, and text documents are all commonly modeled with multinomial distributions. In all of these cases, we expect some form of dependency between the draws: the nucleotide at one position in the DNA strand may depend on the preceding nucleotides, children's names are highly correlated from year to year, and topics in text may be correlated and dynamic. These dependencies are not naturally captured by the typical Dirichlet-multinomial formulation. Here, we leverage a logistic stick-breaking representation and recent innovations in Polya-gamma augmentation to reformulate the multinomial distribution in terms of latent variables with jointly Gaussian likelihoods, enabling us to take advantage of a host of Bayesian inference techniques for Gaussian models with minimal overhead.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, methodology): “Linderman et al. (2015) proposed to use the Pólya-Gamma in a stick-breaking process to reparameterize a multinomial distribution as a product of binomial distributions.”
Efficient Gaussian Process Classification Using Polya-Gamma Data Augmentation (2018, methodology): “1, we compare the approximations to the true posterior obtained by employing an asymptotically correct Gibbs sampler (Polson and Scott 2011; Linderman, Johnson, and Adams 2015).”
Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation (2019, background): “Linderman et al. (2015) consider data augmentation for multinomial likelihoods but focus on sampling.”
Tree-Structured Recurrent Switching Linear Dynamical Systems for Multi-Scale Modeling (2018, background): “Note that this augmentation scheme can also work for negative binomial, binomial, and multinomial observations (Polson et al., 2013; Linderman et al., 2015).”
Bayesian Learning and Inference in Recurrent Switching Linear Dynamical Systems (2017, methodology): “In particular, our main technical contribution is an inference algorithm that leverages Pólya-gamma auxiliary variable methods [Polson, Scott, and Windle, 2013, Linderman, Johnson, and Adams, 2015] to make inference both fast and easy.”