Online Learning for Latent Dirichlet Allocation

By Matthew D. Hoffman, David M. Blei, Francis R. Bach. 2010

1,765 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

GAN Inversion: A Survey (2021, methodology): “For example, we can formulate GAN inversion as decomposing signals into components (matrix factorization problems) and use non-linear Factor Analysis (FA) [208], Independent Component Analysis (ICA) [209], Latent Dirichlet Allocation (LDA) [210], [211] to decompose the network weight and to find int”
Practical Bayesian Optimization of Machine Learning Algorithms (2012, neutral): “We perform a direct comparison between our EI-based approach and GP-UCB in Section 4.1.”
Stochastic variational inference (2012, background): “…factorization models (Spearman, 1904; Tipping and Bishop, 1999; Collins et al., 2002; Wang, 2006; Salakhutdinov and Mnih, 2008; Paisley and Carin, 2009; Hoffman et al., 2010b), certain Bayesian nonparametric mixture models (Antoniak, 1974; Escobar and West, 1995; Teh et al., 2006a), and others.”
Variational Bayesian Inference with Stochastic Search (2012, methodology): “In addition, recent development of the theory has extended the method to online inference and stochastic optimization settings, making variational Bayes a viable approach for Bayesian learning with massive data sets (Hoﬀman et al., 2010; Wang et al., 2011).”
Image-Based Recommendations on Styles and Substitutes (2015, methodology): “ of the products iand j. In other words, we simply adapted our WNN baseline to make use of topic vectors rather than image features.5 We used a 100-dimensional topic model trained using Vowpal Wabbit [8]. However, this baseline proved not to be competitive against the alternatives described above (e”
Efficient Benchmarking of Hyperparameter Optimizers via Surrogates (2015, neutral): “…a logistic regression (Snoek, Larochelle, and Adams 2012) with 4 hyperparameters on the MNIST dataset (LeCun et al. 1998) (both with and without 5-fold cross validation) and an active learning fit of a latent Dirichlet allocation (Hoffman, Blei, and Bach 2010) with 3 continuous hyperparameters.”
Freeze-Thaw Bayesian Optimization (2014, methodology): “Online LDA Next we optimize ﬁve hyperparameters of an online Latent Dirichlet Allocation (LDA) [22] experiment on 250,000 documents from Wikipedia.”
Streaming Variational Bayes (2013, methodology): “The remaining (training) documents C ( train ) are used to estimate the global parameter posterior q ( β ) , and the remaining (training) words W d, train within the d th testing document are used to estimate the document-speciﬁc parameter posterior q ( θ d ) .”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, methodology): “Prior algorithms have used online gradient updates (Duchi et al., 2011) and probabilistic ﬁltering formulations (Murphy, 2002; Hoffman et al., 2010; Broderick et al., 2013).”
Input Warping for Bayesian Optimization of Non-Stationary Functions (2014, methodology): “Experimental setup In this experiment, we apply multi-task warping to logistic regression and online LDA [Hoffman et al., 2010] In the online LDA problem, we assume that a model has been trained on 50 , 000 documents and that we would now like to train one on 200 , 000 documents.”
Automated Variational Inference in Probabilistic Programming (2013, methodology): “This can also be done in an online fashion, in a similar fashion to recent work for stochastic variational inference by Blei et al. [19–21].”
Learning Attitudes and Attributes from Multi-aspect Reviews (2012, methodology): “We acknowledge the existence of more sophisticated variants of LDA, though we are not aware of suitable alternatives that scale to millions of reviews; we used Online LDA as implement in Vowpal Wabbit [15], which required a few hours to train on our largest dataset.”
Hamiltonian Monte Carlo Acceleration Using Neural Network Surrogate functions (2015, background): “One common approach is subsampling (see, for example, [7] [8] [9] [10]), which restricts the computation to a subset of the observed data.”
Towards Bayesian Deep Learning: A Survey (2016, background): “[26] extends LDA from the batch mode to the online setting, making it possible to process large datasets.”
Stochastic Learning on Imbalanced Data: Determinantal Point Processes for Mini-batch Diversification (2017, methodology): “Algorithm 2 BM-SVI We adopt the notation from [13].”
Advances in Variational Inference (2017, background): “Stochastic Variational Inference amounts to applying stochastic optimization to the objective function encountered in VI [65], [67], [71], [209], thereby scaling VI to very large datasets.”
Topic Modeling in Embedding Spaces (2019, methodology): “We use variational inference for all of the models and employ stochastic variational inference (SVI) (Hoffman et al., 2013) to speed up the optimization. The minibatch size is 1,000 documents. For LDA, we set the learning rate as suggested by Hoffman et al. (2013): the delay is 10 and the forgetting”