Distributed Algorithms for Topic Models

By David Newman, Arthur U. Asuncion, Padhraic Smyth, Max Welling. Journal of machine learning research, 2009

469 citations recorded by the source corpus.

Subfields: Artificial Intelligence, General Social Sciences, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/62f991612959e6fff29128b1d1a54596f6504029

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Management Science and Operations Research

Citations in the corpus

A Survey on Distributed Machine Learning (2019, background): “Designing a generic system that enables an ecient distribution of regular Machine Learning is challenging since every algorithm has a distinct communication pattern [78][106][128][146][150][152].”
topicmodels: An R Package for Fitting Topic Models (2011, methodology): “In order to be able to ﬁt topic models to very large data sets distributed algorithms to ﬁt the LDA model were proposed for Gibbs sampling in Newman et al. (2009).”
On Model Parallelization and Scheduling Strategies for Distributed Machine Learning (2014, background): “However, in addition to Big Data, modern large-scale ML problems have started to encounter the so-called Big Model challenge [8, 1, 17], in which models with millions if not billions of parameters and/or variables (such as in deep networks [6] or large-scale topic models [20]) must be estimated from”
KB-LDA: Jointly Learning a Knowledge Base of Hierarchy, Relations, and Facts (2015, methodology): “It has been shown that parallel approaches considerably speed up iterative inference methods such as collapsed Gibbs sampling, resulting in test data log probabilities indistinguishable from those obtained using serial methods (Porte-ous et al., 2008; Newman et al., 2009).”
Stochastic variational inference (2012, methodology): “Despite its popularity in Bayesian inference, relatively little work has focused on developing MCMC algorithms that can scale to very large data sets.”
Asymptotically Exact, Embarrassingly Parallel MCMC (2013, methodology): “To run MCMC when data is partitioned among multiple machines, each machine can perform computation that involves a subset of the data and exchange information at each iteration to draw a sample [10, 14, 18]; however, this requires a significant amount of communication between machines, which can gre”
Topic Modeling Using Latent Dirichlet allocation (2021, methodology): “exploited the inference process by executing in distributed setup, at the same time updating the topic count globally [100].”
Distributed large-scale natural graph factorization (2013, background): “Furthermore, the bulk-synchronous processing paradigm of Map-Reduce [19] does not afford low-enough latency for fast convergence: this has been reported, e.g. in comparisons between bulk-synchronous convergence [20] and asynchronous convergence [23].”
Distributed Multi-Task Relationship Learning (2017, background): “There have been tremendous efforts done on differentmachine learning problems [4, 13, 19].”
Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds (2017, background): “…running across 11 Amazon EC2 global regions and on a cluster that emulates EC2 WAN bandwidth show that Gaia provides 1.8–53.5× speedup over two state-of-the-art distributed ML systems, and is within 0.94–1.40× of the speed of running the same ML algorithm on machines on a local area network (LAN).”
An architecture for parallel topic models (2010, methodology): “More speciﬁcally, the largest published work on LDA by the UC Irvine team [7] is to use 1000 computers for 10 hours to process 8 Million documents taken from PubMed abstracts, which is equivalent to a processing speed of 6,400 documents per computer and hour.”