Gradient Descent Happens in a Tiny Subspace

By Guy Gur-Ari, Daniel A. Roberts, Ethan Dyer. 2018

304 citations recorded by the source corpus.

We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.

Subfields: Artificial Intelligence, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Scaling Laws for Neural Language Models (2020, neutral): “Making this connection quantitative will require a characterization of the Hessian spectrum [Pap18, GKX19, GARD18].”
An Empirical Model of Large-Batch Training (2018, background): “A natural interpretation of these results is that large Hessian directions are dominating the update [GARD18], so that training involves rapid oscillations, as seen in Figure 18 (see [Goh17] for an intuitive picture of these oscillations).”
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, methodology): “Throughout the optimization, we observe that almost all of the gradient energy is concentrated in these outlier subspaces (Figure 11), reproducing an observation of Gur-Ari et al. [12] 7 .”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “Second, Gur-Ari et al. (2019) argue that the SGD trajectory lies in the subspace spanned by the eigenvectors of the Hessian corresponding to the few top eigenvalues.”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “…the eigenspectrum of the Hessian settles into a bulk of small eigenvalues and a few large outlier eigenvalues after some amount of training (Gur-Ari et al., 2018), and networks trained with large batch sizes and high learning rates beneﬁt from learning rate warmup during the ﬁrst part of…”
A Modern Take on the Bias-Variance Tradeoff in Neural Networks (2018, background): “(2017) showed that the spectrum of the Hessian for over-parameterized networks splits into (i) a bulk centered near zero and (ii) a small number of large eigenvalues; and Gur-Ari et al. (2018) recently gave evidence that the small subspace spanned by the Hessian’s top eigenvectors is preserved over ”
The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size. (2018, background): “Gur-Ari et al. [2018] showed empirically that the gradients of SGD are spanned by the top eigenvectors of the Hessian and hinted that projecting the gradients onto this low-dimensional subspace could lead to optimization benefits.”
Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks (2019, background): “[14] Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer.”
Understanding Gradient Clipping in Private SGD: A Geometric Perspective (2020, background): “[2020], Gur-Ari et al. [2018] on deep learning training, and we also provide our own empirical evaluation on the probability term in the Appendix.”
Emergent properties of the local geometry of neural loss landscapes (2019, background): “The majority of the gradient power lies within the subspace spanned by the top few eigenvectors of the Hessian, consistent with observations in real neural networks [12].”
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2024, background): “Subspace Learning Recent studies have demonstrated that the learning primarily occurs within a significantly low-dimensional parameter subspace (Larsen et al., 2022; Gur-Ari et al., 2018).”
Subspace Inference for Bayesian Deep Learning (2019, background): “Relationship to Eigenvalues of the Hessian Li et al. (2018b) and Gur-Ari et al. (2019) argue that the first principal components of the SGD trajectory correspond to the top eigenvectors of the Hessian of the loss, and that these eigenvectors change slowly during training.”
The Early Phase of Neural Network Training (2020, background): “Hessian eigenspectrum: The shape of the loss landscape around the network state also appears to change rapidly during the early phase of training (Sagun et al., 2017; Gur-Ari et al., 2018).”
MetaInit: Initializing learning by learning to initialize (2019, background): “In addition, it has been shown that a signature of failure in deep learning models is when the gradient concentrates on the large eigenvalues of the Hessian [21, 18].”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, unknown): “Language Model Training Dynamics : Previous work has extensively analyzed training dynamics to understand how neural models acquire information over training [1, 30, 34, 66, 74].”
Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification (2020, background): “We take inspiration from the empirical observation from [20, 25, 29] that even though the ambient dimension of the gradients is large, the set of sample gradients at most iterations along the optimization trajectory is often contained in a much lower-dimensional subspace.”