Structured Sparse Principal Component Analysis

By Rodolphe Jenatton, Guillaume Obozinski, Francis R. Bach. International Conference on Artificial Intelligence and Statistics, 2009

362 citations recorded by the source corpus.

We present an extension of sparse PCA, or sparse dictionary learning, where the sparsity patterns of all dictionary elements are structured and constrained to belong to a prespecified set of shapes. This \emph{structured sparse PCA} is based on a structured regularization recently introduced by [1]. While classical sparse priors only deal with \textit{cardinality}, the regularization we use encodes higher-order information about the data. We propose an efficient and simple optimization procedure to solve this problem. Experiments with two practical tasks, face recognition and the study of the dynamics of a protein complex, demonstrate the benefits of the proposed structured approach over unstructured approaches.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/ee1477dcbf768cd589cdcf45ca8f9427666dfe4d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Proximal Methods for Hierarchical Sparse Coding (2010, methodology): “• A reweighted-least-square scheme (Re-ℓ2), as described by Jenatton et al. (2009); Kim and Xing (2010).”
Nuclear Norm Based Matrix Regression with Applications to Face Recognition with Occlusion and Illumination Changes (2014, methodology): “The current methods like LRC, SRC and CRC all use L 2 norm to define the decision rule.”
Online Learning for Matrix Factorization and Sparse Coding (2009, methodology): “However, neither PCA, nor NMF are able to learn localized features on the set of natural patches F.”
Exploiting Geographical Neighborhood Characteristics for Location Recommendation (2014, methodology): “In [12], Jenatton et al. presented an extension of sparse PCA based on a structured regularization that encoded high order information of the input data.”
Optimization with Sparsity-Inducing Penalties (2011, background): “Several instances of the general variational form we considered in this section have appeared in the literature [70, 109, 110].”
Learning Semantic Image Representations at a Large Scale (2014, background): “Various machine learning algorithms aim to solve this, most notably structured sparse PCA [54].”
Bayesian group factor analysis with structured sparsity (2016, background): “For example, in structured sparse PCA, the loading matrix is constrained to have specific patterns (Jenatton et al., 2010).”
Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization (2013, background): “Another related work is the incremental majorization-minimization algorithm of [18] for ﬁnite training sets; it was indeed shown to be efﬁcient for solving machine learning problems where storing”
Factorized Latent Spaces with Structured Sparsity (2010, methodology): “We demonstrate the effectiveness of our approach on the problem of human pose estimation where the existence of shared and private spaces has been shown [7].”
Learning with Submodular Functions: A Convex Optimization Perspective (2011, background): “When applied to sparsity-inducing norms, this leads to supports which are contiguous segments (see applications in [115]).”
Half-Quadratic-Based Iterative Minimization for Robust Sparse Representation (2014, methodology): “In addition, soft-thresholding function and iteratively reweighted methods have drawn much attention in subspace segmentation [11], robust alignment [58], nuclear norm minimization [59], [60], and structure sparsity [61], [62] where the used minimization functions are related to HQ optimization.”
Fast Robust PCA on Graphs (2015, neutral): “Fast Robust PCA on Graphs Nauman Shahid ∗ , Nathanael Perraudin, Vassilis Kalofolias, Gilles Puy † , Pierre Vandergheynst Email: { nauman.shahid, Signal Processing nathanael.perraudin, vassilis.kalofolias, pierre.vandergheynst @epﬂ.ch, † gilles.puy@inria.fr Switzerland Laboratory 2”
Proximal Methods for Sparse Hierarchical Dictionary Learning (2010, methodology): “Structured sparsity has been used to regularize dictionary elements by Jenatton et al. (2010), but to the best of our knowledge, it has never been used to model dependencies between them.”
Structured Variable Selection with Sparsity-Inducing Norms (2009, background): “, 2009) and sparse matrix factorization problems (Mairal et al., 2009; Jenatton et al., 2009).”
Sparse Modeling for Image and Vision Processing (2014, neutral): “More generally, for any q ∈ (0, 2), and ‖α‖q = (∑q j=1 |α[j]|q )1/q and r = q/(2 − q), we have [Jenatton et al., 2010b]: ‖α‖q = inf η∈Rp+ 1 2 p∑ j=1 α[j]2 η[j] + 1 2‖η‖r, with the optimal η in Rp+ obtained as η[j] = |α[j]|2−q‖α‖q−1q .”
Recent advances in feature selection and its applications (2017, background): “Solution 1 In order to remedy the deficiency of Lasso and due to the importance of structural information, such as group, recently a general definition of the structured sparsity-inducing norm is proposed to incorporate the prior knowledge or structural constraints to find the suitable linear featur”