Measuring the Intrinsic Dimension of Objective Landscapes

By Chunyuan Li, Heerad Farkhoor, Rosanne Liu, Jason Yosinski. 2018

566 citations recorded by the source corpus.

Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem. But how accurate are such notions? How many parameters are really needed? In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace. We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape. The approach is simple to implement, computationally tractable, and produces several suggestive conclusions. Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes. This latter result has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold. Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10. In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution. A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.

Subfields: Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “This is in accordance with the ideas of Li et al. [18], who hypothesize that the loss surface has low intrinsic dimensionality, and also with results of Sagun et al. on toy models.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, positive): “Lastly, our experimental results empirically support that, mere parameter counting does not always correlate with the effective capacity of the models at scale [85, 86].”
A Neural Scaling Law from the Dimension of the Data Manifold (2020, background): “The intrinsic dimension of the neural network parameter space has also been discussed [LFLY18].”
LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “We take inspiration from [1, 20] which show that the learned over-parametrized models in fact reside on a low intrinsic dimension.”
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “(2020) have attempted to explain the empirical effectiveness of transfer from pruned networks by the notion of intrinsic-dimensionality (Li et al., 2018): the minimum dimension that is needed to optimize a given objective to some precision.”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “Low-rank parameterization Low-rank methods [13, 14] have demonstrated that strong performance can be achieved by optimizing a task in a low-rank subspace.”
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020, methodology): “Unfortunately, the method of ﬁnding the intrinsic dimension proposed by Li et al. (2018) is un-aware of the layer-wise structure of the function parameterized by θ .”
Weight Agnostic Neural Networks (2019, background): “Recent works offer a modern treatment [6] and application [61, 108] of these principles in the context of larger, deep neural network architectures.”
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (2018, methodology): “We use subspace training (Li et al., 2018) to measure the model complexity in text classification problems.”
Joint Embedding of Words and Labels for Text Classification (2018, methodology): “(2018a) quantitatively show that the word-embeddings-based text classification tasks can have the similar level of difficulty regardless of the employed models, using the concept of intrinsic dimension (Li et al., 2018).”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, background): “For example, Li et al. (2018) ﬁnd that the “intrinsic” dimensionality of popular architectures can be on the order of hundreds to a few thousand.”
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2018, methodology): “If we assume that we are in fact detecting analogs of support vectors, we can put these results in perspective with the intrinsic dataset dimension defined by Li et al. (2018) as the codimension in the parameter space of the solution set: for a given architecture, the higher the intrinsic dataset di”
Superposition of many models into one (2019, background): “Li et al. (2018) found the intrinsic dimensionality of most tasks to be a small fraction of the dimensionality of the pa-B rameters.”
The Description Length of Deep Learning models (2018, methodology): “For 2-part and network compression codes, we report results from (Han et al., 2015a) and (Xu et al., 2017), and for the intrinsic dimension code, results from (Li et al., 2018).”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, background): “…sizes, e.g. to make training more stable as in unitary evolution RNNs (Arjovsky et al., 2015) and in weight normalization (Salimans & Kingma, 2016), to inject inductive biases (Thomas et al., 2018), and to alter (Dinh et al., 2017) or to measure (Li et al., 2018) properties of the loss landscape.”
An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution (2018, methodology): “The parsimony of parameters further confirms they are simply more appropriate models for the task of coordinate transform [28, 10, 19].”
Deep Ensembles: A Loss Landscape Perspective (2019, background): “Li et al. (2018) demonstrated that constraining optimization to a random, low-dimensional hyperplane in the weight space leads to results comparable to full-space optimization, provided that the dimension exceeds a modest threshold.”
Robust fine-tuning of zero-shot models (2021, background): “[59] observe that solution spaces for a given task are high dimensional, while D’Amour et al.”