Statistical inference for model parameters in stochastic gradient descent

By Xi Chen, Jason D. Lee, Xin T. Tong, Yichen Zhang. 2016

169 citations recorded by the source corpus.

The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistical inference of true model parameters based on SGD when the population loss function is strongly convex and satisfies certain smoothness conditions. Our main contributions are two-fold. First, in the fixed dimension setup, we propose two consistent estimators of the asymptotic covariance of the average iterate from SGD: (1) a plug-in estimator, and (2) a batch-means estimator, which is computationally more efficient and only uses the iterates from SGD. Both proposed estimators allow us to construct asymptotically exact confidence intervals and hypothesis tests. Second, for high-dimensional linear regression, using a variant of the SGD algorithm, we construct a debiased estimator of each regression coefficient that is asymptotically normal. This gives a one-pass algorithm for computing both the sparse regression coefficients and confidence intervals, which is computationally attractive and applicable to online data.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “[6] then show that using A and the sample average of the iterates for a Gaussian approximation produces well calibrated confidence intervals of the parameters and that the variance of these estimators achieves the Cramer Rao lower bound (the minimum possible variance).”
Uncertainty Estimation with Infinitesimal Jackknife, Its Distribution and Mean-Field Approximation (2020, methodology): “The trajectory of stochastic gradient descent gives rise to a sequence of models where the covariance matrix among batch means converges to H − 1 JH − 1 [9, 10, 33], similar to the pseudo-ensemble distributions in form.”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, background): “II, models aleatoric uncertainty by introducing additional output layers to the detection head for estimating the variance of output bounding boxes [8], [9], [11]–[17], [31], [80], [81], [83], [87]–[91] or the variance of output logits [12], [87].”
Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach (2016, methodology): “In the second vein of providing confidence regions for the solution x itself, several researchers propose using the iterates of a stochastic approximation algorithm [22, 25, 68, 66].”
Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints (2017, background): “Algorithm-dependent generalization bounds and statistical properties have been intensively studied for SGM under convex settings (Hardt et al., 2015; Lin and Rosasco, 2016; Lin et al., 2016; Wei et al., 2017; Chen et al., 2016), but very few are known for the non-convex case.”
Bridging the gap between constant step size stochastic gradient descent and Markov chains (2017, background): “Moreover, convergence results can be used to derive confidence intervals for θ∗, as in [13, 57].”
Fast Uncertainty Estimates and Bayesian Model Averaging of DNNs (2018, methodology): “Directly computing this covariance would be quite expensive; however, [3] proposed using a version of the sample covariance, Σ, as an estimator of this matrix, thus justifying our usage of the sample covariance of SWA.”