Randomized Nonlinear Component Analysis

By David Lopez-Paz, Suvrit Sra, Alex Smola, Zoubin Ghahramani, Bernhard Schölkopf. 2014

185 citations recorded by the source corpus.

Classical methods such as Principal Component Analysis (PCA) and Canonical Correlation Analysis (CCA) are ubiquitous in statistics. However, these techniques are only able to reveal linear relationships in data. Although nonlinear variants of PCA and CCA have been proposed, these are computationally prohibitive in the large scale. In a separate strand of recent research, randomized methods have been proposed to construct features that help reveal nonlinear patterns in data. For basic tasks such as regression or classification, random features exhibit little or no loss in performance, while achieving drastic savings in computational requirements. In this paper we leverage randomness to design scalable new variants of nonlinear PCA and CCA; our ideas extend to key multivariate analysis tools such as spectral clustering or LDA. We demonstrate our algorithms through experiments on real-world data, on which we compare against the state-of-the-art. A simple R implementation of the presented algorithms is provided.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Unifying distillation and privileged information (2015, methodology): “After distillation, we can use the simpler f s ∈ F s for faster prediction at test time.”
Scalable Kernel Methods via Doubly Stochastic Gradients (2014, background): “Thus the required number of random feature (or ranks), r, will be of the order O(n) = O(1/ ) [4, 5, 8, 9].”
Large-scale kernel methods for independence testing (2016, methodology): “For completeness, the proof of this theorem, which is a consequence of Lyons (2013, Theorem 2.7) and the equivalence between distance-based and RKHS-based independence statistics (Sejdinovic et al. 2013b) is given in Appendix. As remarked by Sejdinovic et al. (2013b), the finite marginal moment cond”
A Survey of Multi-View Representation Learning (2016, unknown): “Consequently, a deep canonically correlated autoencoder (DCCAE) [20] is introduced through jointly optimizing the objectives of the deep CCA and the separate autoencoders from multiple views.”
Nonparametric Canonical Correlation Analysis (2015, methodology): “Several approximate techniques have been proposed, largely based on low-rank kernel matrix approximations [Bach and Jordan, 2002, Hardoon et al., 2004, Arora and Livescu, 2012, Lopez-Paz et al., 2014].”
Scale Up Nonlinear Component Analysis with Doubly Stochastic Gradients (2015, methodology): “Recently, methods such as Randomized Component Analysis (RCA) [12] are able to scale to larger datasets by leveraging random feature approximation.”
On Deep Multi-View Representation Learning (2015, methodology): “…on a randomly shufﬂed dataset with careful tuning of kernel widths for FKCCA/NKCCA (using rank M = 6000 ) and obtained canonical correlations of 99.2/105.6/107.6 for FKCCA/NKCCA/DCCA, which are better than those reported in Lopez-Paz et al. (2014). same objective as STO given more training time.”
From Dependence to Causation (2016, methodology): “To extend the PCA algorithm to discover nonlinear principal components, while avoiding the computational burden of KPCA, we propose Randomized PCA or RPCA (Lopez-Paz et al., 2014).”