Solving large scale linear prediction problems using stochastic gradient descent algorithms

By Tong Zhang. 2004

1,250 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, neutral): “Distributed learning is also considered in [16, 49, 57, 78, 87, 89].”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “This problem has received attention from various communities over the last fifty years in optimization, stochastic approximation, and machine learning (Polyak and Juditsky, 1992; Ruppert, 1988; Fabian, 1968; Nesterov and Vial, 2008; Nemirovski et al., 2009; Shalev-Shwartz et al., 2009; Zhang, 2004).”
Protection Against Reconstruction and Its Applications in Private Federated Learning (2018, positive): “Stochastic gradient methods are now the de facto approach to large-scale model-ﬁtting [68, 18, 60, 28], and recent work of McMahan et al. [54] describes systems (which they term federated learning ) for aggregating multiple stochastic model-updates from distributed mobile devices.”
Adaptive Bound Optimization for Online Convex Optimization (2010, methodology): “2 The practical importance of adaptive regularization In the past few years, online algorithms have emerged as state-of-the-art techniques for solving large-scale machine learning problems (Bottou & Bousquet, 2008; Zhang, 2004).”
DeltaGrad: Rapid retraining of machine learning models (2020, background): “More recently a lot of work (see e.g., Bottou, 1998; 2003; Zhang, 2004; Bousquet & Bottou, 2008; Bottou, 2010; Bottou et al., 2018) focuses on empirical risk minimization.”
Stochastic dual coordinate ascent methods for regularized loss (2012, neutral): “A simple approach for solving SVM is stochastic gradient descent (SGD) [16, 13, 1].”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, methodology): “The following are two ways to choose gt, like SGD [41, 4, 34] and SVRG [15] SGD : gt(wt) = ∇fnt(wt), SVRG : gt(wt) = ∇fnt(wt)−∇fnt(w̃) +∇f(w̃) (2) where nt is uniformly sampled from the data set and w̃ is a reference point.”
Parallel SGD: When does averaging help? (2016, background): “Stochastic gradient descent (SGD) Shalev-Shwartz et al. [2009], Zhang [2004], Zinkevich [2003] is the workhorse of modern machine learning, largely due to its simplicity and scalability to very large datasets Bottou [2010].”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, methodology): “Stochastic gradient descent is an active ﬁeld (Zhang, 2004; Bottou, 1998).”
Mini-Batch Primal and Dual Methods for SVMs (2013, methodology): “In the context of SVMs, approaches of this form include primal stochastic gradient descent (SGD) methods (e.g., Pegasos, Shalev-Shwartz et al. 2011, NORMA, Zhang 2004) and dual stochastic coordinate ascent (Hsieh et al., 2008).”
Machine Learning with Membership Privacy using Adversarial Regularization (2018, methodology): “For solving the optimization problem (3), especially for non-convex loss functions for complex models such as deep neural networks, the commonly used method is the stochastic gradient descent algorithm [4, 52].”
Semi-Stochastic Gradient Descent Methods (2013, background): “Unlike gradient descent, stochastic gradient descent [1, 2] instead picks a random i (uniformly) and updates xk+1 = xk − hf ′ i (xk).”
Stochastic Optimization with Importance Sampling for Regularized Loss Minimization (2014, methodology): “Stochastic optimization has been extensively studied in the machine learning community [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
Katyusha: the first direct acceleration of stochastic gradient methods (2016, background): “…satisfying E [ ∇ k ] = ∇ f ( x k ) and is referred to as the gradient estimator . e e e A popular choice for the gradient estimator is to set ∇ k = ∇ f i ( x k ) for some random index i ∈ [ n ] per iteration, and methods based on this choice are known as stochastic gradient descent (SGD) [10, 37].”
Asynchrony begets momentum, with an application to deep learning (2016, background): “(1) Typically the component function fi(w) represents a loss function evaluated on a specific data point zi.”
Byzantine-Tolerant Machine Learning (2017, methodology): “In most SGD-based learning algorithms used today [3,5,6,25], the choice function consists in computing the average of the input vectors.”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, methodology): “For more information about modern dual and primal methods we refer the reader to the literature on randomized coordinate descent methods [129, 148, 151, 122, 59, 163, 111, 121, 149, 57, 140, 36] and stochastic gradient methods [156, 196, 105, 79, 172, 132, 140, 155], respectively.”