Scalable inference in latent variable models

By Amr Ahmed, Mohamed Aly, Joseph E. Gonzalez, Shravan M. Narayanamurthy, Alex Smola. 2012

281 citations recorded by the source corpus.

Latent variable techniques are pivotal in tasks ranging from predicting user click patterns and targeting ads to organizing the news and managing user generated content. Latent variable techniques like topic modeling, clustering, and subspace estimation provide substantial insight into the latent structure of complex data with little or no external guidance making them ideal for reasoning about large-scale, rapidly evolving datasets. Unfortunately, due to the data dependencies and global state introduced by latent variables and the iterative nature of latent variable inference, latent-variable techniques are often prohibitively expensive to apply to large-scale, streaming datasets. In this paper we present a scalable parallel framework for efficient inference in latent variable models over streaming web-scale data. Our framework addresses three key challenges: 1) synchronizing the global state which includes global latent variables (e.g., cluster centers and dictionaries); 2) efficiently storing and retrieving the large local state which includes the data-points and their corresponding latent variables (e.g., cluster membership); and 3) sequentially incorporating streaming data (e.g., the news). We address these challenges by introducing: 1) a novel delta-based aggregation system with a bandwidth-efficient communication protocol; 2) schedule-aware out-of-core storage; and 3) approximate forward sampling to rapidly incorporate new data. We demonstrate state-of-the-art performance of our framework by easily tackling datasets two orders of magnitude larger than those addressed by the current state-of-the-art. Furthermore, we provide an optimized and easily customizable open-source implementation of the framework1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, methodology): “Some MCMC methods can also exploit these innovations (Welling and Teh 2011; Ahmed et al. 2012).”
Deep Factors for Forecasting (2019, methodology): “For instance, in Temporal LDA (Ahmed et al., 2012) we assume that we have a common ﬁxed Dirichlet-Multinomial distribution capturing the distribution of tokens per topic, and a time-variant set of latent random variables capturing the changes in user preferences.”
More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server (2013, methodology): “idea of a parameter server [22, 2], which we define as the combination of a shared key-value store that provides a centralized storage model (which may be implemented in a distributed fashion) with a synchronization model for reading/updating model values.”
Petuum: A New Platform for Distributed Machine Learning on Big Data (2013, methodology): “Platform Comparison Figure 9 (left) compares Petuum to popular ML platforms (Spark and GraphLab) and well-known cluster implementations (YahooLDA [2]).”
Clipper: A Low-Latency Online Prediction Serving System (2016, background): “Parameter Server: There has been considerable work in the learning systems community on parameterservers [5, 21, 38, 62].”
Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey (2017, unknown): “Yahoo LDA [1] C++ Gibbsampling https://github.”
Parameter Server for Distributed Machine Learning (2013, methodology): “The globally shared word-to-topic assignments are then maintained by server nodes [26, 2] while the clients send state change updates to the server.”
Fast Differentially Private Matrix Factorization (2015, methodology): “The performance improvement over existing libraries such as GraphChi are due to both cache efficient design, prefetching, pipelining, the fact that we exploit the power law property of the data, and by judicious optimization of random number generation.”
GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server (2016, methodology): “Figure 14 compares the per-mini-batch AdamLike model training time with 8 machines, when using GeePS, GeePSsingle-table, and CPU-PS, and each of three synchronization models: BSP (“Slack 0”), SSP (“Slack 1”), and Asynchronous (“Slack Inf”).”
WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation (2015, methodology): “Following the previous work [1, 32], we measure the model quality by the widely adopted log joint likelihood (log likelihood in short):”
Latent factor models with additive and hierarchically-smoothed user preferences (2013, methodology): “In the future we plan to incorporate the dynamics of user interests [4] and scale our inference to hundred of millions of users using techniques from [3]”
Communication Efficient Distributed Machine Learning with the Parameter Server (2014, background): “There are also open source codes, such as YahooLDA [2] and Petuum [15].”
High-Performance Distributed ML at Scale through Parameter Server Consistency Models (2014, background): “minibatch methods) or randomized algorithms, which are prominently represented in the large-scale ML literature (Ahmed et al. 2012; Ho et al. 2013; Gemulla et al. 2011; Kumar et al. 2014).”
Addressing the straggler problem for iterative convergent parallel ML (2016, methodology): “State-of-the-art frameworks for high-performance parallel ML use a parameter server architecture [1, 3, 11, 12, 15, 16, 29, 35], which does not accommodate computation redundancy.”
Deep Learning for Time Series Forecasting: Tutorial and Literature Survey (2020, background): “In [3, 123], other combined global and local models are detailed.”
Distributed Bayesian Learning with Stochastic Natural Gradient Expectation Propagation and the Posterior Server (2015, methodology): “Note however that our algorithmic contribution is effectively orthogonal to (Ahmed et al., 2012), who proposed a generic and robust computational architecture for distributed machine learning.”
Launchpad: A Programming Model for Distributed Machine Learning Research (2021, background): “Parameter servers (Ahmed et al., 2012; Li et al., 2014) are an approach for addressing the storage and updating of parameters in distributed machine learning systems.”