An architecture for parallel topic models

By Alex Smola, Shravan M. Narayanamurthy. 2010

469 citations recorded by the source corpus.

This paper describes a high performance sampling architecture for inference of latent topic models on a cluster of workstations. Our system is faster than previous work by over an order of magnitude and it is capable of dealing with hundreds of millions of documents and thousands of topics. The algorithm relies on a novel communication structure, namely the use of a distributed (key, value) storage for synchronizing the sampler state between computers. Our architecture entirely obviates the need for separate computation and synchronization phases. Instead, disk, CPU, and network are used simultaneously to achieve high performance. We show that this architecture is entirely general and that it can be extended easily to more sophisticated latent variable models such as n-grams and hierarchies.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

In ML Systems & Hardware

Citations in the corpus

TensorFlow: A system for large-scale machine learning (2016, neutral): “Parameter servers emerged as an architecture for scalable topic modeling [66], and our previous system DistBe-lief [21] showed how a similar architecture could be applied to deep neural network training.”
Stochastic variational inference (2012, methodology): “Finally, in topic modeling, researchers have developed several approaches to parallel MCMC (Newman et al., 2009; Smola and Narayanamurthy, 2010; Ahmed et al., 2012).”
Decentralized gradient methods: does topology matter? (2020, neutral): “In the first case, a stateful parameter server (PS) (Smola and Narayanamurthy, 2010) maintains the current version of the model parameters.”
Distributed GraphLab: A Framework for Machine Learning in the Cloud (2012, background): “Nonetheless, the demand for large-scale computational and storage resources, has driven many [2] [14] [15] [27] [30] [35] to develop new parallel and distributed MLDM systems targeted at individual models and applications.”
Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study (2015, background): “[22] describes a typical parameter server using a distributed key-value store to synchronize state between processes.”
Multiple domain user personalization (2011, methodology): “We use the approach of [18] for distributed collapsed inference and build on this framework.”
Scaling Distributed Machine Learning with the Parameter Server (2014, unknown): “Since its introduction, the parameter server framework [43] has proliferated in academia and industry.”
Online Machine Learning in Big Data Streams (2018, methodology): “ good performance, there must be sufﬁcient inherent parallelism in the modeling approach. Fig. 2 The parameter server architecture for distributed machine learning. The parameter server introduced in [209] is a popular way to implement model parallel training, with several variants developed [115, 1”
Topical Word Embeddings (2015, methodology): “From the complexity analysis, we can see that, compared with Skip-Gram, TWE models require more parameters to record more discriminative information for word embeddings, but the computational complexity does not increase too much relatively, especially when many fast algorithms of topic models are a”
PowerLyra: Differentiated Graph Computation and Partitioning on Skewed Graphs (2019, background): “Graph-structured computation has be come increasingly popular, which is evidenced by its adoption in a wide range of areas including social computation, web search, natural language processing, and recommendation systems [5, 25, 54, 67, 88, 98].”
Communication Efficient Distributed Machine Learning with the Parameter Server (2014, methodology): “The system offers two relaxations to address data (in)consistency: First, rather than arguing for a specific consistency model [29, 7, 15], we support flexible consistency models.”
TF-Replicator: Distributed Machine Learning for Researchers (2019, methodology): “Subsequent systems have extended the parameter server model, introduced by (Smola & Narayanamurthy, 2010) and later popularized by (Li et al., 2013; 2014) for machine learning.”
Proposal Scaling Distributed Machine Learning with System and Algorithm Co-design (2016, unknown): “In [123] a system with eventual consistency is discussed.”
A Machine Learning Approach to Twitter User Classification (2011, methodology): “The model is obtained by training a LDA parallel implementation (Smola and Narayanamurthy 2010) with 500 iterations over a set of 4M users, each represented by a maximum of 20,000 words collected from their tweets.”
Distributed GraphLab: A Framework for Machine Learning in the Cloud (2012, background): “tate, and communication protocols while simultaneously developing mathematically complex models and algorithms. Nonetheless, the demand for large-scale computational and storageresources,hasdrivenmany[2,14,15,27,30,35]todevelopnew parallel and distributed MLDM systems targeted at individual modelsan”
Distributed large-scale natural graph factorization (2013, background): “Furthermore, the bulk-synchronous processing paradigm of Map-Reduce [19] does not afford low-enough latency for fast convergence: this has been reported, e.g. in comparisons between bulk-synchronous convergence [20] and asynchronous convergence [23].”
GaDei: On Scale-Up Training as a Service for Deep Learning (2016, methodology): “ts fault-tolerance. 7.Related work Deep learning has seen tremendous success in image recognition [35], natural language processing [23], speech translation [15] and gaming [49]. The parameter server [51] based approach is the de facto method to scale out such training tasks in a distributed environ”
Topic Modeling Using Latent Dirichlet allocation (2021, background): “Smola and Narayanamurthy proposed a general architecture for inferring topics over multiple workstations in parallel mode [127].”