Intrinsic dimension of data representations in deep neural networks

By Alessio Ansuini, Alessandro Laio, Jakob H. Macke, Davide Zoccolan. 2019

425 citations recorded by the source corpus.

Deep neural networks progressively transform their inputs across multiple processing layers. What are the geometrical properties of the representations learned by these networks? Here we study the intrinsic dimensionality (ID) of data-representations, i.e. the minimal number of parameters needed to describe a representation. We find that, in a trained network, the ID is orders of magnitude smaller than the number of units in each layer. Across layers, the ID first increases and then progressively decreases in the final layers. Remarkably, the ID of the last hidden layer predicts classification accuracy on the test set. These results can neither be found by linear dimensionality estimates (e.g., with principal component analysis), nor in representations that had been artificially linearized. They are neither found in untrained networks, nor in networks that are trained on randomized labels. This suggests that neural networks that can generalize are those that transform the data into low-dimensional, but not necessarily flat manifolds.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Geometry and Topology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Geometry and Topology

In Model Architectures

Citations in the corpus

A Neural Scaling Law from the Dimension of the Data Manifold (2020, methodology): “5 (for the error rate) [RRBS19] and a smaller ID ≈ 8 [ALMZ19].”
A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, background): “A possible reason is that shallow layers tend to provide generalpurpose representations, whereas later layers specialize [35, 3, 4, 56, 59].”
Patch SVDD: Patch-level SVDD for Anomaly Detection and Segmentation (2020, background): “The ID is the minimal number of coordinates required to describe the points without significant information loss [32].”
The Intrinsic Dimension of Images and Its Impact on Learning (2021, background): “Ansuini et al. (2019) further found that the intrinsic dimension of features decreases in late layers of neural networks and observed interesting trends in the dimension of features in early layers.”
Geometric compression of invariant manifolds in neural nets (2020, background): “Still, more robust measures, such as the effective dimension of the neural representation of the data, support that compression occurs in deeper layers [7, 8].”
Why Do Better Loss Functions Lead to Less Transferable Features? (2020, background): “More generally, studies have investigated how class information evolves through the hidden layers of neural networks using linear classifiers [2], binning estimators of mutual information [61, 58, 26], Euclidean distances [59], the soft nearest neighbor loss [24], and manifold geometry [3, 16].”
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings (2025, unknown): “In high-dimensional spaces, the ratio of the distances to the first and second nearest neighbors follows a Pareto distribution for points uniformly distributed on a d -dimensional manifold (Ansuini et al., 2019).”