Constant Step Size Stochastic Gradient Descent for Probabilistic Modeling
By Dmitry Babichev, Francis R. Bach. 2018
11 citations recorded by the source corpus.
Stochastic gradient methods enable learning probabilistic models from large amounts of data. While large step-sizes (learning rates) have shown to be best for least-squares (e.g., Gaussian noise) once combined with parameter averaging, these are not leading to convergent algorithms in general. In this paper, we consider generalized linear models, that is, conditional models based on exponential families. We propose averaging moment parameters instead of natural parameters for constant-step-size stochastic gradient descent. For finite-dimensional models, we show that this can sometimes (and surprisingly) lead to better predictions than the best linear model. For infinite-dimensional models, we show that it always converges to optimal predictions, while averaging natural parameters never does. We illustrate our findings with simulations on synthetic data and classical benchmarks with many observations.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Andrew Gordon Wilson
- Pavel Izmailov
- Dmitry P. Vetrov
- Wesley J. Maddox
- Timur Garipov
- Stanislav Fort
- Matthew D. Hoffman
- Max Welling
Citations in the corpus
- A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “[10], and Babichev and Bach [3] all use the same style of analyses, but for different purposes.”