Non-negative Matrix Factorization with Sparseness Constraints

By Patrik O. Hoyer. Journal of machine learning research, 2004

3,063 citations recorded by the source corpus.

Non-negative matrix factorization (NMF) is a recently developed technique for finding parts-based, linear representations of non-negative data. Although it has successfully been applied in several applications, it does not always result in parts-based representations. In this paper, we show how explicitly incorporating the notion of 'sparseness' improves the found decompositions. Additionally, we provide complete MATLAB code both for standard NMF and for our extension. Our hope is that this will further the application of these methods to solving novel data-analysis problems.

Subfields: Atomic and Molecular Physics, and Optics, Cognitive Neuroscience, Signal Processing

View Paper: https://www.semanticscholar.org/paper/ede3af3637977988b8cbb330c294183e919b7d5a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atomic and Molecular Physics, and Optics

In Cognitive Neuroscience

In Signal Processing

Citations in the corpus

Rectified Factor Networks (2015, background): “away by noise. The second problem is that sparse priors like Laplacians for the coding units are computationally very expensive when calculating the code for a new input (Olshausen & Field, 1996; Hoyer, 2004; Rozell et al., 2008; Gregor & LeCun, 2010). The representation of the input is the ”
A survey of dimensionality reduction techniques (2014, methodology): “NMF with Sparseness Constraints [Hoyer2004] performs the NMF with the constraint that the sparseness of each column of W is a given constant w S (i.e., it promotes the sparseness of the dictionary atoms) and the sparseness of each row of U is another constant U S (i.e., it promotes that each atom…”
The Why and How of Nonnegative Matrix Factorization (2014, background): “The most popular prior is sparsity which can be tackled with projections [64] or with ℓ 1 -norm penalty [72, 48].”
Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference (2020, methodology): “Our sparsity-boosting methods combine a regularizer following the Hoyer sparsity metric (Hoyer, 2004), together with a variant of the classic ReLU activation, which we call Forced Activation Threshold ReLU (FATReLU).”
Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis (2007, methodology): “NMF generated holistic basis images instead of parts-based basis images for a facial image dataset in the results presented in Li et al. (2001) and Hoyer (2004).”
Online Learning for Matrix Factorization and Sparse Coding (2009, positive): “Following this line of research, Hoyer (2002, 2004) has proposed non negative sparse coding (NNSC), which extends non-negative matrix factorization by adding a sparsity-inducing penalty to the objective function to further control the sparsity of the vectors α i : min When λ = 0, this formulation…”
SLIM: Sparse Linear Methods for Top-N Recommender Systems (2011, methodology): “The computational complexity can be potentially reduced by utilizing the sparse matrix factorization algorithms developed in [16], [17], [18].”
Hyperspectral Unmixing via $L_{1/2}$ Sparsity-Constrained Nonnegative Matrix Factorization (2011, methodology): “In general, the L1 regularizer is the most popular choice for achieving sparsity of the abundance matrix [26]–[28].”
Nonnegative Matrix Factorization: A Comprehensive Review (2013, methodology): “These two objective functions are usually adopted, such as NSC [89], [90] and NMFSC [86]; meanwhile, some more effective approaches are proposed to resolve the optimization problems.”
Generative-Discriminative Basis Learning for Medical Imaging (2012, methodology): “In addition to the nonnegativity and sparsity constraints used in previous work [23], [24], we introduce a new type of constraint (group-sparsity) that allows further anatomical coupling between voxels defined by a segmentation (II-D).”
Proximal alternating linearized minimization for nonconvex and nonsmooth problems (2013, methodology): “Moreover, extended versions of NMF with sparsity were considered via relaxations and corresponding convex re-formulations solved by sophisticated and computationally demanding conic programming schemes, see e.g., [20,21].”
FABIA: factor analysis for bicluster acquisition (2010, methodology): “Following (Hoyer, 2004), we define sparseness as”
DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures (2019, methodology): “In early non-negative matrix factorization (NMF) research, a consensus was that a sparsity measure should map a ndimensional vector to a real number between 0 and 1, where the sparsest possible vectors (with only one nonzero element) have sparsity 1 and the vectors with all elements equal have spars”
Nonnegative Least-Correlated Component Analysis for Separation of Dependent Sources by Volume Maximization (2010, background): “[ 19 ]. Possible circumstances under which the NMF yields unique decomposition can also be found in [20].”
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification (2020, methodology): “…scaling problem of the L 1 regularizer, previous works in compressed sensing have been using Hoyer regularizer to induce sparsity in solving non-negative matrix factorization [11] and blind deconvolution [16], where the Hoyer regularizer shows superior performance comparing to other methods.”
Sparse Filtering (2011, methodology): “In particular, sparse filtering resembles the `1 `2 sparsity penalty that has been used in non-negative matrix factorization [35].”
RECONSTRUCTING THE STELLAR MASS DISTRIBUTIONS OF GALAXIES USING S4G IRAC 3.6 AND 4.5 μm IMAGES. I. CORRECTING FOR CONTAMINATION BY POLYCYCLIC AROMATIC HYDROCARBONS, HOT DUST, AND INTERMEDIATE-AGE STARS (2011, background): “Although sparseness constraints on NMF (e.g. NMFSC; Hoyer 2004) have been shown to improve the uniqueness of solutions in some applications, neither the sparseness of the sources nor their ‘mixing’ coefficients can be specified individually.”
Projected Gradient Methods for Nonnegative Matrix Factorization (2007, methodology): “As penalty parameters α and β only indirectly control the sparseness, Hoyer (2004) proposes a scheme to directly specify the desired sparsity.”