Winner-Take-All Autoencoders

By Alireza Makhzani, Brendan J. Frey. 2014

254 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Large Memory Layers with Product Keys (2019, neutral): “In winner take all autoencoders [29], sparsity is induced by using mini-batch statistics, while in the sparse access memory [35] reports some speed-up by both thresholding the memory to a sparse subset, and by using efﬁcient data structures for content-based read operations.”
Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification (2016, methodology): “With pooling-induced sparsity bottlenecks (Makhzani & Frey, 2015), the convolutional SAE (Masci et al., 2011) can learn features from middle-size images.”
A Survey of Inductive Biases for Factorial Representation-Learning (2016, methodology): “We can incorporate the bias of a sparse representation into an autoencoder architecture by adding a sparsity penalty [37] to get a sparse autoencoder.”
The Utility of Sparse Representations for Control in Reinforcement Learning (2018, methodology): “Winnner-Take-All auto-encoders (Makhzani and Frey 2015) use a k% response constraint per node across instances, during training, to pro-1”
Image compression with Stochastic Winner-Take-All Auto-Encoder (2017, unknown): “Note that [14] uses WTA, but our WTA rule is different and gα does not apply to specific dimensions of its input tensor as this constraint is not relevant for image compression.”
Image reconstruction by domain-transform manifold learning (2017, methodology): “The convolutional layers are inspired by Winner-Take-All autoencoders 37 which jointly optimize the sparse convolutional codes as well as the deconvolutional kernel “dictionaries” upon which the final image is built (Fig.”
Top-Down Learning for Structured Labeling with Convolutional Pseudoprior (2015, background): “[33]), but none of them work in our case.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “…algorithm [218] or L 0 -norm constrained gradient descent [219]. b) Neuron pruning: Neurons are also prunable by using the simple magnitude-aware rule (e.g., for A [237] or for E [202], [233]), according to the neuronal selectivity [238] or even based on random [239]/probabilistic [240] selection.”
Deep One-Class Classification (2018, unknown): “…(Makhzani & Frey, 2013), variational autoencoders (VAEs) (Kingma & Welling, 2013), and deep convolutional autoencoders (DCAEs) (Masci et al., 2011; Makhzani & Frey, 2015), where the last variant is predominantly used in AD applications with image or video data (Seeböck et al., 2016; Richter &…”
Neural Networks and Deep Learning (2018, unknown): “It is not necessary to tie the weights in the encoder and decoder, or even use a symmetric architecture between encoder and decoder [310].”