Learning with SGD and Random Features

By Luigi Carratino, Alessandro Rudi, Lorenzo Rosasco. 2018

87 citations recorded by the source corpus.

Sketching and stochastic gradient methods are arguably the most common techniques to derive efficient large scale learning algorithms. In this paper, we investigate their application in the context of nonparametric statistical learning. More precisely, we study the estimator defined by stochastic gradient with mini batches and random features. The latter can be seen as form of nonlinear sketching and used to define approximate kernel methods. The considered estimator is not explicitly penalized/constrained and regularization is implicit. Indeed, our study highlights how different parameters, such as number of features, iterations, step-size and mini-batch size control the learning properties of the solutions. We do this by deriving optimal finite sample bounds, under standard assumptions. The obtained results are corroborated and illustrated by numerical experiments.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Kernel Continual Learning (2021, methodology): “To address this, we further introduce random Fourier features to learn data-driven kernels, which have previously demonstrated success in regular learning tasks (Bach et al., 2004; Sinha & Duchi, 2016; Carratino et al., 2018; Zhen et al., 2020).”
On Lazy Training in Differentiable Programming (2018, methodology): “For two-layer neural networks, we show in Appendix A.2 that the linearized model is a random feature model [26] which lends itself nicely to statistical analysis [6].”
Optimal Statistical Rates for Decentralised Non-Parametric Regression with Linear Speed-Up (2019, background): “1 We note, while this assumes agents communicate infinite dimensional quantities in the general nonparametric setting, the framework we consider accommodates finite approximations of infinite dimensional quantities whilst accounting for the statistical precision [13].”
Learning to Learn Kernels with Variational Random Features (2020, unknown): “Kernel approximation by random Fourier features (RFFs) (Rahimi & Recht, 2007) is an effective technique for efﬁcient kernel learning (Gärt-ner et al., 2002), which has recently become increasingly popular (Sinha & Duchi, 2016; Carratino et al., 2018).”