Augment and Reduce: Stochastic Inference for Large Categorical Distributions

By Francisco J. R. Ruiz, Michalis K. Titsias, Adji Bousso Dieng, David M. Blei. 2018

22 citations recorded by the source corpus.

Categorical distributions are ubiquitous in machine learning, e.g., in classification, language models, and recommendation systems. They are also at the core of discrete choice models. However, when the number of possible outcomes is very large, using categorical distributions becomes computationally expensive, as the complexity scales linearly with the number of outcomes. To address this problem, we propose augment and reduce (A&R), a method to alleviate the computational complexity. A&R uses two ideas: latent variable augmentation and stochastic variational inference. It maximizes a lower bound on the marginal likelihood of the data. Unlike existing methods which are specific to softmax, A&R is more general and is amenable to other categorical models, such as multinomial probit. On several large-scale classification problems, we show that A&R provides a tighter bound on the marginal likelihood and has better predictive performance than existing approaches.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Adversarial Contrastive Estimation (2018, methodology): “…that the computational cost could potentially be reduced via some existing techniques such as the “augment and reduce” variational inference of (Ruiz et al., 2018), adaptive softmax (Grave et al., 2016), or the “sparsely-gated” softmax of Shazeer et al. (2017), but leave that to future work.”
Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation (2019, methodology): “Using large categorical distributions is a challenging problem (Ruiz et al., 2018; Titsias, 2016).”