Variance Reduction for Stochastic Gradient Optimization

By Chong Wang, X. Chen, Alex Smola, Eric P. Xing. 2013

161 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, background): “Other work proposes variants of semi-stochastic algorithms to approximate the exact gradient direction and reduce the gradient variance [47, 34].”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “the size of variance) can be lower-bounded as [36]”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, methodology): “From Figures 4, we observe that the proposed sparsification approach is at least comparable to QSGD, and significantly outperforms QSGD when the gradient sparsity is stronger; and this concords with our analysis on the gradient approximate sparsity encouraging faster speed up.”
Measuring and regularizing networks in function space (2018, background): “…that many approaches to normalize and whiten activations or gradients have been forwarded in the literature (Raiko et al. (2012);Simard et al. (1998); Schraudolph & Sejnowski (1996); Crammer et al. (2009); Wang et al. (2013); LeCun et al. (1991); Schraudolph (1998); Salimans & Kingma (2016)).”
Momentum-Based Variance Reduction in Non-Convex SGD (2019, unknown): “[27].”
Black-box Importance Sampling (2016, background): “…an importance role, such as probabilistic inference in graphical model (e.g., Liu et al., 2015), variance reduction for variational inference (e.g., Wang et al., 2013; Ranganath et al., 2014) and policy gradient (e.g., Greensmith et al., 2004), covariance shift in transfer learning (e.g., Sugiyama…”
Stochastic Learning on Imbalanced Data: Determinantal Point Processes for Mini-batch Diversification (2017, background): “Several ways of variance reduction of stochastic gradient algorithms have been proposed, an important class relying on control variates [26, 31, 34, 43, 37].”
Minimizing finite sums with the stochastic average gradient (2013, background): “If the step-size 8Wang et al. [2013] consider a related algorithm that maintains an easily-computable approximation to the current gradient g′(xk).”
Balanced Mini-batch Sampling for SGD Using Determinantal Point Processes (2017, background): “Several ways of variance reduction of stochastic gradient algorithms have been proposed, an important class relying on control variates [26, 32, 35, 44, 38].”
Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling (2014, methodology): “To explicitly reduce the variance, some new stochastic gradient algorithms have been proposed [3, 10, 12].”
Efficient Distributed SGD with Variance Reduction (2015, methodology): “Note, the stored gradients and the average gradient term g̃y are initialized using a single epoch of “vanilla” SGD with no VR correction.”
Stochastic quasi-gradient methods: variance reduction via Jacobian sketching (2018, methodology): “Here we handle this issue using control variates, a commonly used tool to reduce variance in Monte Carlo methods [13] and introduced in [35] for designing variance reduced stochastic gradient algorithm.”
Fine-Tuning Language Models with Just Forward Passes (2023, background): “[93] Chong Wang, Xi Chen, Alexander J Smola, and Eric P Xing.”
Advances in Variational Inference (2017, background): “ates for variance reduction is common in Monte Carlo simulation and stochastic optimization [165], [208].”
Active Sampler: Light-weight Accelerator for Complex Data Analytics at Scale (2015, background): “It has been shown by [27, 11, 23] that simply reducing the variance of stochastic gradient will increase the convergence rate of SGD algorithms.”
A Linearly-Convergent Stochastic L-BFGS Algorithm (2015, methodology): “For a partial list, see (Kingma and Ba, 2015; Sutskever et al., 2013; Duchi et al., 2011; Shalev-Shwartz and Zhang, 2013; Johnson and Zhang, 2013; Roux et al., 2012; Wang et al., 2013; Nesterov, 2009; Frostig et al., 2015; Agarwal et al., 2014).”
Black Box Variational Inference (2013, background): “, 2012) and for general optimization problems that occur in machine learning (Wang et al., 2013).”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “ion using the pathwise and score function estimators have been explored in several papers including Kucukelbir et al. (2017); Miller et al. (2017); Paisley et al. (2012); Salimans and Knowles (2013); Wang et al. (2013). Decreasing variance over training. Figure 8 shows the variance of the two estima”
Stochastic Gradient Made Stable: A Manifold Propagation Approach for Large-Scale Optimization (2015, background): “Most of prior works [12], [15] report the performance with respect to iteration counts.”