Non-negative sparse coding

By Patrik O. Hoyer. 2002

927 citations recorded by the source corpus.

Non-negative sparse coding is a method for decomposing multivariate data into non-negative sparse components. We briefly describe the motivation behind this type of data representation and its relation to standard sparse coding and non-negative matrix factorization. We then give a simple yet efficient multiplicative algorithm for finding the optimal values of the hidden components. In addition, we show how the basis vectors can be learned from the observed data. Simulations demonstrate the effectiveness of the proposed method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “There are many existing theoretical works studying algorithm for sparse coding [9, 17, 43, 47, 54, 68, 85, 91, 94], however, these algorithms share little similarity to training a neural network.”
Sparse Overcomplete Word Vector Representations (2015, methodology): “To obtain nonnegative sparse word vectors, we use a variation of the nonnegative sparse coding method (Hoyer, 2002).”
Online Learning for Matrix Factorization and Sparse Coding (2009, positive): “Following this line of research, Hoyer (2002, 2004) has proposed non negative sparse coding (NNSC), which extends non-negative matrix factorization by adding a sparsity-inducing penalty to the objective function to further control the sparsity of the vectors α i : min When λ = 0, this formulation…”
Sparse Representation Based Fisher Discrimination Dictionary Learning for Image Classification (2014, background): “Apart from the l0- or l1-norm sparsity penalty, nonnegative (Hoyer 2002), group (Bengio et al.”
Graph Regularized Nonnegative Matrix Factorization for Data Representation (2011, background): “Recent studies show that NMF does not always result in parts-based representations [22], [23].”
Document clustering based on non-negative matrix factorization (2003, neutral): “NMF is a matrix factorization algorithm that ﬁnds the positive factorization of a given positive matrix [7, 8, 6].”
Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding (2012, methodology): “We find that such word representations are effective in tasks of interest, are sparse, and are also interpretable, when learned by Non-Negative Sparse Embedding (NNSE) – a variation on Non-Negative Sparse Coding, which is a matrix factorization technique previously studied in the machine learning co”
Document clustering using nonnegative matrix factorization (2006, methodology): “The method proposed by Hoyer [6] has an important feature that enforces a statistical sparsity of the H matrix.”
Generalized Nonnegative Matrix Approximations with Bregman Divergences (2005, neutral): “DivergenceDφ φ α β Remarks ‖A−BC‖(2)F 12x2 0 0 Lee and Seung [11, 12] ‖A−BC‖(2)F 12x2 0 λ1 C1 Hoyer [10] ‖W (A−BC)‖(2)F 12x2 0 0 Paatero and Tapper [13] KL(A,BC) x log x− x 0 0 Lee and Seung [11] KL(A,WBC) x log x− x 0 0 Guillamet et al.”
On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization (2007, background): “DISCUSSION AND CONCLUSIONS Earlier work such as [11] and [12] adds penalty terms to increase the sparsity of and”
Online Nonnegative Matrix Factorization With Robust Stochastic Approximation (2012, background): “Hoyer [26] proposed incorporating the l1-regularization on the coefficients.”
Projected Gradient Methods for Nonnegative Matrix Factorization (2007, methodology): “We consider three image problems used in Hoyer (2004): The CBCL (MIT Center for Biological and Computional Learning) face image database (http://cbcl.”
Nonsmooth nonnegative matrix factorization (nsNMF) (2006, methodology): “Combining a small reconstruction error with a sparseness criterion, the objective function defined in [ 15 ] is:”
Algorithms and applications for approximate nonnegative matrix factorization (2007, methodology): “The notion of sparsity refers sometimes to a representational scheme where only a few features are effectively used to represent data vectors (Hoyer, 2002, 2004).”
K-SVD and its non-negative variant for dictionary design (2005, methodology): “from which all training signals are built of [15, 16].”
SVD based initialization: A head start for nonnegative matrix factorization (2008, background): “Another important issue addressed by researchers is the incorporation of further constraints appropriate for the problem; see e.g. [6] [7] [13] [15].”