The effective rank: A measure of effective dimensionality

By Olivier Roy, Martin Vetterli. European Signal Processing Conference, 2007

559 citations recorded by the source corpus.

Subfields: Computational Mechanics, Statistical and Nonlinear Physics, Signal Processing

View Paper: https://www.semanticscholar.org/paper/497eb64754cace3afd79aab6062fddf8ca21b60f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Statistical and Nonlinear Physics

In Signal Processing

Citations in the corpus

Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, unknown): “We find that the tuned lens resolves the problems with the logit lens discussed in Section 2: it has significantly lower 3 Pythia and GPT-NeoX-20B were trained using the same architecture, data, and codebase (Andonian et al., 2021).”
Muon is Scalable for LLM Training (2025, unknown): “For a weight matrix with singular values σ = ( σ 1 , σ 2 , · · · , σ n ) , we calculate the SVD entropy (Alter et al. 2000; Roy et al. 2007) of this matrix as follows: As shown in Figure 4, we visualized the average SVD entropy of the weight matrices across different training checkpoints during…”
Loss of plasticity in deep continual learning (2024, unknown): “Similar to the rank of a matrix, which represents the number of linearly independent dimensions, the effective rank takes into consideration how each dimension influences the transformation induced by a matrix [47].”
Implicit Regularization via Neural Feature Alignment (2020, methodology): “We quantify this using a notion of eﬀective rank based on spectral entropy (Roy & Vetterli, 2007).”
Reconfigurable Intelligent Surfaces for Wireless Communications: Principles, Challenges, and Opportunities (2020, unknown): “The spectral entropy [62], [63] of an M × N matrix X, denoted by SE(X), is given by”
Fair and Diverse DPP-based Data Summarization (2018, background): “Such a notion of low effective rank appears frequently in the machine learning literature [24, 9].”
The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2022, background): “Effective rank was introduced by Roy and Vetterli (2007) in the context of signal processing; the effective rank of a matrix is deﬁned as the exponential of the entropy of the normalized singular values.”
Implicit Regularization in Deep Learning May Not Be Explainable by Norms (2020, background): “The discrete nature of rank renders its direct analysis unfavorable from a dynamical perspective (the rank of a matrix implies little about its proximity to low-rank), thus we consider the following surrogate measures: (i) effective rank (Definition 1 below; from [74]) — a continuous extension of ra”
Implicit Regularization in Deep Matrix Factorization (2019, background): “[43]), in a manner which is stronger for deeper factorizations.”
Clinical Context Variables Collectively Rival Model Choice in Embedding-Based Retrieval: Multi-Corpus Benchmark Study (2026, unknown): “Effective rank was computed as the exponential of the Shannon entropy of the normalized singular value spectrum of the embedding matrix, measuring the effective dimensionality used by the model [27].”