A fast parallel SGD for matrix factorization in shared memory systems

By Yong Zhuang, Wei-Sheng Chin, Yu-Chin Juan, Chih-Jen Lin. 2013

167 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

In Optimization & Training

Citations in the corpus

On Sampling Strategies for Neural Network-based Collaborative Filtering (2017, methodology): “Sampling techniques are also studied in [9, 39] to distribute the computation of matrix factorization, their objectives in sampling strategy design are reducing the parameter overlapping and cache miss.”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “There has also been work in parallelizing stochastic optimization algorithms for specific problems (like PCA) in the past, for e.g., Mcdonald et al. (2009); McDonald et al. (2010); Meng et al. (2012); Zhuang et al. (2013); Li et al. (2014a); Chin et al. (2015); Oh et al. (2015).”
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent (2017, methodology): “…in solving large-scale machine learning problems such as deep learning [Dean et al., 2012, Li et al., 2014], matrix completion [Recht et al., 2011, Zhuang et al., 2013] and SVM. Existing PSGD algorithms are mostly designed for centralized network topology, for example, parameter server [Li et…”
Recent advances in convolutional neural networks (2015, methodology): “[70] also uses multiple GPUs to asynchronously calculate gradients and update the global model parameters, which achieves 3.2 times of speedup on 4 GPUs compared to training on a single GPU.”
Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time (2017, methodology): “CF relies on factorizing user-item interaction matrices to generate latent factors representing users and items [16, 17, 25, 30].”
Perturbed Iterate Analysis for Asynchronous Stochastic Optimization (2015, background): “A large body of recent work has demonstrated that near-linear speedups are achievable, in theory and practice, on many common machine learning tasks [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 13, 16].”
Guaranteed Matrix Completion via Non-Convex Factorization (2014, background): “Index Terms —Matrix completion, matrix factorization, nonconvex optimization, alternating minimization, SGD, perturbation analysis.”
Cyclades: Conflict-free Asynchronous Machine Learning (2016, methodology): “…study parallel asynchronous iterative algorithms on the computational model used by [NRRW11], and similar to the partially asynchronous model of [BT89]: a number of cores have access to the same shared memory, and each of them can read and update components of x in parallel from the shared memory.”
A Fast Parallel Stochastic Gradient Method for Matrix Factorization in Shared Memory Systems (2015, background): “A preliminary version of this work appears in a conference paper [Zhuang et al. 2013].”
Partially Observable Markov Decision Process for Recommender Systems (2016, methodology): “In order to make future recommendations, we generate the candidates by varying the parameters of a matrix factorization model [30], and the candidate with the highest Q value is adopted for the future recommendation.”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, background): “A considerable amount of effort (Zinkevich et al., 2010; Recht et al., 2011; Dean et al., 2012; Zhuang et al., 2013; Paine et al., 2013; Yadan, Adams, Taigman, & Ranzato, 2014; Krizhevsky, 2014) has gone in to parallelizing the training of DCNNs via model parallelism, which entails the use of GPUs,…”
NOMAD: Nonlocking, stOchastic Multi-machine algorithm for Asynchronous and Decentralized matrix completion (2013, methodology): “All algorithms were initialized with the same initial parameters; we set each entry of W and H by independently sampling a uniformly random variable in the range (0 , 1 √ k ) [26, 28].”
A Fast Distributed Stochastic Gradient Descent Algorithm for Matrix Factorization (2014, methodology): “In order to settle the locking problem that DSGD faces in DCE, based on DSGD Gemulla et al. (2011) and FPSGD Zhuang et al. (2013), we put forward a fast distributed SGD algorithm called FDSGD.”
Fast and Robust Parallel SGD Matrix Factorization (2015, methodology): “A random order of SGD updating is important for fast convergence [5, 25], thus most other SGD algorithms (e.g., DSGD, FPSGD, NO-MAD) are designed with random order of SGD updates.”
A Harmonic Extension Approach for Collaborative Ranking (2016, result): “An important direction is to further improve the efficiency of the algorithm and compare it with recent large-scale matrix completion methods (Yun et al., 2014; Zhuang et al., 2013).”