Averaged Least-Mean-Squares: Bias-Variance Trade-offs and Optimal Sampling Distributions

By Alexandre D'efossez, Francis R. Bach. 2015

88 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Several studies focus on understanding properties of SGD for quadratic functions: a detailed nonasymptotic analysis is provided by Défossez and Bach (2015), acceleration under the additive noise oracle (see Assumption A4 below) is studied by Dieuleveut et al. (2016) (without this assumption by Jain…”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, background): “Recent studies investigate the merits of adaptive step sizes (Duchi et al., 2011; Tieleman and Hinton, 2012), gradient or iterate averaging (Toulis et al., 2016; Défossez and Bach, 2015), and constant step-sizes (Bach and Moulines, 2013; Flammarion and Bach, 2015).”
A Variational Analysis of Stochastic Gradient Algorithms (2016, background): “Recent studies investigate the merits of adaptive step sizes (Duchi et al., 2011; Tieleman and Hinton, 2012), gradient or iterate averaging (Toulis et al.; D´efossez and Bach, 2015), and constant step-sizes (Bach and Moulines, 2013; Flammarion and Bach, 2015).”
The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure (2019, methodology): “With regards to least squares, we assume equation (3), following recent efforts Bach and Moulines (2013); D ´ efossez and Bach (2015); Jain et al. (2016).”
Accelerating Stochastic Gradient Descent (2017, methodology): “We use the operator viewpoint for analyzing stochastic gradient methods, introduced in Défossez and Bach (2015).”
Parallelizing Stochastic Gradient Descent for Least Squares Regression: Mini-batching, Averaging, and Model Misspecification (2016, methodology): “The tool employed in obtaining these results generalizes the operator view of averaged SGD with batch size 1 (Défossez and Bach, 2015) and a clear exposition of the bias-variance decomposition from Jain et al.”
Communication trade-offs for Local-SGD with large step size (2019, background): “…studies focus on understanding properties of SGD for quadratic functions: a detailed non-asymptotic analysis is provided by Défossez and Bach [28], acceleration under the additive noise oracle (see Assumption A 4 below) is studied by Dieuleveut et al. [29] (without this assumption by Jain et…”
Batched Stochastic Gradient Descent with Weighted Sampling (2016, background): “Other papers exploring advantages of importance sampling in various adaptations of stochastic gradient descent include but are not limited to [19, 20, 21, 22].”
Complexity Analysis of stochastic gradient methods for PDE-constrained optimal Control Problems with uncertain parameters (2021, methodology): “These Stochastic Gradient (SG) techniques have been extensively applied to machine learning problems [14,16,19,33], but have not yet been used much for PDE-constrained optimization under uncertainty.”
AdaBatch: Efficient Gradient Aggregation Rules for Sequential and Parallel Stochastic Gradient Methods (2017, background): “In practical applications, the bias term can be the most important one to optimize for [23].”
Parallelizing Stochastic Approximation Through Mini-Batching and Tail-Averaging (2016, methodology): “We note that such a characterization automatically sheds light on how the maximal divergent learning rates interpolate batch sizes of 1 (which is ≤ H [1]) to the batch gradient descent learning rate (setting b to ∞ ), which turns out to be 2 λ max ( H ) .”
Bridging the gap between constant step size stochastic gradient descent and Markov chains (2017, background): “This last term was improved to ‖Σ−1/2θ∗‖2/(γn)2 by [15, 16], showing that asymptotically, the bias term is negligible, see also [31].”