Scalable Coordinate Descent Approaches to Parallel Matrix Factorization for Recommender Systems

By Hsiang-Fu Yu, Cho-Jui Hsieh, Si Si, Inderjit S. Dhillon. 2012

294 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Recommendations as Treatments: Debiasing Learning and Evaluation (2016, methodology): “This means that we can readily draw upon the existing arsenal of optimization algorithms (i.e., Gemulla et al., 2011; Yu et al., 2012), making the training problem highly scalable.”
Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms (2015, background): “HOGWILD! and other lock-free algorithms have been applied to a variety of uses, including PageRank approximations (FrogWild! [16]), deep learning (Dogwild! [18]) and recommender systems [24].”
Petuum: A New Platform for Distributed Machine Learning on Big Data (2013, methodology): “Some of these properties have been successfully used in bespoke, large-scale implementations of popular ML algorithms: topic models [31, 19], matrix factorization [30, 16], and deep learning [17].”
Guaranteed Matrix Completion via Non-Convex Factorization (2014, background): “Index Terms —Matrix completion, matrix factorization, nonconvex optimization, alternating minimization, SGD, perturbation analysis.”
Parallel coordinate descent methods for big data optimization (2012, background): “Several papers on the topic appeared around the time this paper was finalized or after [6,14,22,22,28].”
A Generic Coordinate Descent Framework for Learning from Implicit Feedback (2016, background): “For MF, [23] shows a complexity of O ( | S | k ) and for FM, [11] derives a complexity of O ( N Z ( X ) k ) where N Z ( X ) is the number of non-zero entries in the design matrix X .”
Logistic Matrix Factorization for Implicit Feedback Data (2014, background): “Additionally, MF models have been shown to be efficiently parallelizable and highly scalable to large web scale datasets [7, 19, 23, 24].”
A Fast Parallel Stochastic Gradient Method for Matrix Factorization in Shared Memory Systems (2015, methodology): “Although [Yu et al. 2012] have considered MovieLens, we use a diﬀerent setting of λ P = λ Q = 0 .”
Pseudo-Relevance Feedback Based on Matrix Factorization (2016, methodology): “[39] proposed a parallel algorithm to improve the efficiency of the NMF algorithm.”
Recommender Systems (2016, unknown): “Other methods such as coordinate descent can effectively address the trade-off between efficiency and stability [650].”
NOMAD: Nonlocking, stOchastic Multi-machine algorithm for Asynchronous and Decentralized matrix completion (2013, methodology): “This is relatively simple for ALS [27] and CCD++ [26], but a bit more involved for SGD [12, 18].”
A Fast Distributed Stochastic Gradient Descent Algorithm for Matrix Factorization (2014, methodology): “Algorithm 1 DSGD’s Process.”
Let's Make Block Coordinate Descent Go Fast: Faster Greedy Rules, Message-Passing, Active-Set Complexity, and Superlinear Convergence (2017, background): “Other scenarios where coordinate descent and BCD methods have proven useful include matrix and tensor factorization methods [98, 96], problems involving log-determinants [82, 36], and problems involving convex extensions of sub-modular functions [37, 22].”
High-Performance Distributed ML at Scale through Parameter Server Consistency Models (2014, unknown): “2013), or special-purpose solvers tailored to specific categories of ML applications: CCD++ (Yu et al. 2012) for Matrix Factorization, Vowpal Wabbit for regression/classification problems via stochastic optimization (Langford, Li, and Strehl 2007), and Yahoo LDA as well as Google plda for topic mode”
PASSCoDe: Parallel ASynchronous Stochastic dual Co-ordinate Descent (2015, methodology): “is in computing wTx i, which requires O(nnz=n)time. For SVM problems, the subproblem has a closed form solution, while for logistic regression problems it has to be solved by an iterative solver (see Yu et al. (2012) for details). The DCD algorithm, which is part of the popular LIBLINEAR package, is”
On Model Parallelization and Scheduling Strategies for Distributed Machine Learning (2014, methodology): “We solve this objective using a parallel coordinate descent algorithm [24].”
A fast parallel SGD for matrix factorization in shared memory systems (2013, methodology): “From empirical experiences, the influence of the initial value of P and Q cannot be ignored, so we carefully set these values to be identical to [13].”
Fast and Robust Parallel SGD Matrix Factorization (2015, methodology): “Efﬁcient algorithms for matrix factorization have been developed including alternating least square (ALS) [9, 24], coordinate descent [8, 22], and stochastic gradient descent (SGD) [26, 5].”
Primal-dual algorithms for non-negative matrix factorization with the Kullback-Leibler divergence (2014, result): “Similar optimization approaches were previously reported in [15, 16].”