Parallel stochastic gradient algorithms for large-scale matrix completion

By B. Recht, Christopher R'e. 2013

378 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mathematics, Computational Mechanics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mathematics

In Computational Mechanics

In Optimization & Training

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “However, the attention computation may be par-4 allelizable across multiple GPUs [75].”
Parallel SGD: When does averaging help? (2016, background): “As a result, there has been a lot of interest in parallelizing SGD Delalleau and Bengio [2007], Recht and Ré [2013], Recht et al. [2011], Zinkevich et al. [2010].”
Non-stochastic Best Arm Identification and Hyperparameter Optimization (2015, methodology): “Collaborative filtering We next consider a matrix completion problem using the Movielens 100k dataset trained with stochastic gradient descent on the biconvex objective with step sizes as described in [25].”
Memory Bounded Deep Convolutional Networks (2014, background): “See [17] for discussion specifically on projection within stochastic gradient methods.”
Speeding Up Distributed Machine Learning Using Codes (2015, background): “to randomly re-shuffle the data a number of times [29], [61]–[65].”
Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System (2019, background): “Furthermore, unlike ALS, parallelization of SGD is challenging, and numerous approaches have been proposed to distribute the learning of matrix factorization [14, 15, 16, 17].”
On the rates of convergence of parallelized averaged stochastic gradient algorithms (2017, background): “In recent work, Recht and Ré [2] also deals with stochastic gradient by proposing a rewriting procedure where each processor can rewrite the data of an other.”
Communication-Efficient Edge AI: Algorithms and Systems (2020, background): “• Data reshuffling via index coding and pliable index coding: Data reshuffling [202], [203] is a recognized approach to improve the statistical performance of machine learning algorithms.”
Distributed optimization for multi-task learning via nuclear-norm approximation (2015, methodology): “…of nuclear norm regularization described above have inspired research in parallel computation following the model of stochastic gradient descent (Recht and Ré, 2013), but these developments emphasize the parallel aspect alone, rather than other aspects such as geographically distributed data,…”
Large-scale matrix factorization with distributed stochastic gradient descent (2011, background): “DSGD can handle a wide variety of matrix factorizations.”
Permutation-Based SGD: Is Random Optimal? (2021, methodology): “Although permutation-based SGD has been widely observed to perform better in practice (Bottou, 2009; Recht & Ré, 2012; 2013), the vanilla version has attracted the vast majority of theoretical analysis.”
An Overview of Low-Rank Matrix Recovery From Incomplete Observations (2016, background): “rm, other proxies for rank exist. The max-norm [127,128] is a convex function that results in a similar optimization program to (7), and is subject to similar heuristics as (10) for storage reduction [117]. Alternatively, the logarithm of the determinant is a nonconvex, but smooth, proxy for rank wh”
Scalable Coordinate Descent Approaches to Parallel Matrix Factorization for Recommender Systems (2012, methodology): “For matrix factorization, [5] and [6] propose Distributed SGD (DSGD)2 which partitions A into blocks and updates a set of independent blocks in parallel at the same time.”