k-Sparse Autoencoders

By Alireza Makhzani, Brendan J. Frey, Brendan J. Frey. 2013

587 citations recorded by the source corpus.

Recently, it has been observed that when representations are learnt in a way that encourages sparsity, improved performance is obtained on classification tasks. These methods involve combinations of activation functions, sampling steps and different kinds of penalties. To investigate the effectiveness of sparsity by itself, we propose the k-sparse autoencoder, which is an autoencoder with linear activation function, where in hidden layers only the k highest activities are kept. When applied to the MNIST and NORB datasets, we find that this method achieves better classification results than denoising autoencoders, networks trained with dropout, and RBMs. k-sparse autoencoders are simple to train and the encoding stage is very fast, making them well-suited to large problem sizes, where conventional sparse coding algorithms cannot be applied.

Subfields: Artificial Intelligence, Computational Mechanics, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Large Memory Layers with Product Keys (2019, background): “For instance, the k-sparse autoencoder [28] only keeps the k largest values in the latent representation of an auto-encoder, similar to our memory layer but without the product keys component.”
Example Selection For Dictionary Learning (2014, methodology): “An even simpler variant of the pursuit-type algorithm is the thresholding Elad (2010) or the k-Sparse algorithm Makhzani and Frey (2013).”
Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks (2016, background): “On the other hand, it is generally observed that useful filters that constitute a good representation of real world data are roughly incoherent (Wright et al., 2010; Makhzani & Frey, 2013); thus ensuring the R.H.S is small thereby minimizing the error bound.”
The Utility of Sparse Representations for Control in Reinforcement Learning (2018, methodology): “…on the activation (denoted by (cid:96) 1 R-NN and (cid:96) 2 R-NN respectively); k-sparse NNs, where all but the top k activations are zeroed (Makhzani and Frey 2013) (k-sparse-NN); and Winner-Take-All NNs that We include learning curves and instance sparsity for these methods, for a ReLU…”
Winner-Take-All Autoencoders (2014, background): “5(f) shows the deep feature maps of a stacked CONV-WTA that was trained on MNIST.”
Scaling and evaluating sparse autoencoders (2024, unknown): “Makhzani and Frey [2013] refined this concept by introducing k -sparse autoencoders, which use a TopK activation function instead of the L 1 penalty.”
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing (2018, background): “SparseCoding Sparsity isadesirablecharacteristic for representing data as not only it is an efficientmodel of data representation, but also captures the generationprocess ofmost real-world data, in particular textual data [7, 30].”
A survey of deep neural network architectures and their applications (2017, background): “Furthermore, Makhzani and Frey [110] proposed a k-sparse AE in 2013.”
Neural Networks and Deep Learning (2018, unknown): “This approach is referred to as the r-sparse autoencoder [309].”
MARTA GANs: Unsupervised Representation Learning for Remote Sensing Image Classification (2016, unknown): “Several unsupervised representation learning algorithms have been based on the autoencoder [7], [8], which receives corrupted data as input and is trained to predict the original, uncorrupted input.”
On the link between conscious function and general intelligence in humans and machines (2022, background): “That said, there are also other approaches to creating attention mechanisms, such as “hard attention” (Schmidhuber & Huber, 1991; Xu et al., 2015), and top-k sparsity (Makhzani & Frey, 2013; Ahmad & Scheinkman, 2019).”
A Survey on Deep Learning for Multimodal Data Fusion (2020, background): “However, they neglect the spatial and temporal topologies of the multimodal data. by Ngiam et al. (2011) is the most representative deep learning model based on the stacked autoencoder (SAE) for multimodal data fusion.”
Deep Semi-Supervised Anomaly Detection (2019, background): “Choices for regularization include sparsity [40], the distance to some prior latent distribution, e.g. measured via the KL divergence [33, 55], an adversarial loss [41], or simply a bottleneck in dimensionality.”
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing (2018, unknown): “SparseCoding Sparsity isadesirablecharacteristic for representing data as not only it is an efficientmodel of data representation, but also captures the generationprocess ofmost real-world data, in particular textual data [7, 30].”
The Rhetoric and Reality of Anthropomorphism in Artificial Intelligence (2019, methodology): “Other sparsity-inducing methods are commonly deployed in DNNs to prevent against overfitting and stabilize the hidden layers of unsupervised autoencoders (Lee et al. 2008; Makhzani and Frey 2013).”
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, unknown): “We can edit the set of features that models have access to by ablating sparse features that humans deem spurious; we find that this is significantly more effective than a neuron-based ablation method which has an unfair advantage.”
Transform invariant auto-encoder (2017, methodology): “The auto-encoder method [1], [2], [3] is a type of dimensionality reduction method.”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, unknown): “AEs, for example, can be understood from an information-theoretic perspective [556] as adhering to the Infomax principle [557]–[559] by implicitly maximizing the mutual information between the input and latent code—subject to structural constraints or regularization of the code (e.g., “bottleneck,” ”