Linear dimensionality reduction: survey, insights, and generalizations

By John P. Cunningham, Zoubin Ghahramani. 2014

644 citations recorded by the source corpus.

Linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data, due to their simple geometric interpretations and typically attractive computational properties. These methods capture many data features of interest, such as covariance, dynamical structure, correlation between data sets, input-output relationships, and margin between data classes. Methods have been developed with a variety of names and motivations in many fields, and perhaps as a result the connections between all these methods have not been highlighted. Here we survey methods from this disparate literature as optimization programs over matrix manifolds. We discuss principal component analysis, factor analysis, linear multidimensional scaling, Fisher's linear discriminant analysis, canonical correlations analysis, maximum autocorrelation factors, slow feature analysis, sufficient dimensionality reduction, undercomplete independent component analysis, linear regression, distance metric learning, and more. This optimization framework gives insight to some rarely discussed shortcomings of well-known methods, such as the suboptimality of certain eigenvector solutions. Modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver, which accepts as input data and an objective to be optimized, and returns, as output, an optimal low-dimensional projection of the data. This simple optimization framework further allows straightforward generalizations and novel variants of classical methods, which we demonstrate here by creating an orthogonal-projection canonical correlations analysis. More broadly, this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox, objective-agnostic numerical technology.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Bayesian group factor analysis with structured sparsity (2016, background): “Indeed, the field of latent variable models is extremely broad, and robust unifying frameworks are desirable (Cunningham and Ghahramani, 2015).”
Overview and comparative study of dimensionality reduction techniques for high dimensional data (2020, neutral): “Linear Technique [19] Represents each reduced dimension as a linear combination of the original dimension.”
Optimization Models for Machine Learning: A Survey (2019, background): “While this section focuses on Principle Component Analysis and Partial Least Squares, we note that other linear and nonlinear dimension reduction methods exist and an extensive survey with discussions on their benefits and shortcomings is presented in [75].”
Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions (2016, neutral): “sublinear scaling with the size of datasets3. Furthermore, we review tensor decompositions and the associated algorithms for very large-scale linear/multilinear dimensionality reduction problems (see [66, 99, 208, 269, 292, 303] and references therein). The related optimization problems often involv”
Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives (2017, methodology): “This problem can be formulated as the standard symmetric eigenvalue decomposition (EVD), in the form A xk = λkxk, k = 1, 2, . . . , K, (3.21) where xk P RI are the orthonormal eigenvectors and λk the corresponding eigenvalues of a symmetric matrix A P RIˆI .”
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks (2017, methodology): “Various shallow dimensional reduction methods have been unified under the optimization framework with orthogonality constraints (Cunningham and Ghahramani 2015).”
Ten quick tips for effective dimensionality reduction (2019, neutral): “By reducing the dimensionality of the data, you can often alleviate this challenging and troublesome phenomenon.”
Learning in high-dimensional multimedia data: the state of the art (2015, methodology): “LDR [9] interprets linear dimensionality reduction in a simple optimization framework as a program with a problem-speciﬁc objective over orthogonal or unconstrained matrices.”
Conceptual and empirical comparison of dimensionality reduction algorithms (PCA, KPCA, LDA, MDS, SVD, LLE, ISOMAP, LE, ICA, t-SNE) (2021, neutral): “, 65, 165-172 (2016) [17] Cunningham, J.”
From Dependence to Causation (2016, methodology): “…canonical correlations analysis, maximum autocorrelation factors, slow feature analysis, sufficient dimensionality reduction, undercomplete independent component analysis, linear regression, and distance metric learning”, all of these eloquently reviewed in (Cunningham and Ghahramani, 2015).”