A Winner-Take-All Method for Training Sparse Convolutional Autoencoders
By Alireza Makhzani, Brendan J. Frey. 2014
33 citations recorded by the source corpus.
We explore combining the benefits of convolutional architectures and autoencoders for learning deep representations in an unsupervised manner. A major challenge is to achieve appropriate sparsity among hidden variables, since neighbouring variables in each feature map tend to be highly correlated and a suppression mechanism is therefore needed. Previously, deconvolutional networks and convolutional predictive sparse decomposition have been used to construct systems that have a recognition pathway and a data generation pathway that are trained so that they agree and so that the hidden representation is sparse. We take a more direct approach and describe a way to train convolutional autoencoders layer by layer, where in each layer sparsity is achieved using a winner-take-all activation function within each feature map. Learning is computationally efficient and we show that our method can be used to train shallow and deep convolutional autoencoders whose representations can be used to achieve classification rates on the MNIST, CIFAR-10 and NORB datasets that are competitive with the state of the art.
Subfields: Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Ming-Hsuan Yang
- Andrea Vedaldi
- Andrew Zisserman
In Generative Models
- Yann LeCun
- Yoshua Bengio
- Junbo Jake Zhao
- Michaël Mathieu
- Kevin Swersky
- Richard S. Zemel
- Ayman S. El-Baz
- Ehsan Hosseini-Asl
Citations in the corpus
- Generative Moment Matching Networks (2015, background): “For example, it may be possible to use a GMMN+AE with convolutional auto-encoders (Zeiler et al., 2010; Masci et al., 2011; Makhzani & Frey, 2014) in order to create generative models of high resolution color images.”
- Stacked What-Where Auto-encoders (2015, background): “Convolutional auto-encoder variants also relate to our work [10, 18, 19].”
- Alzheimer's disease diagnostics by adaptation of 3D convolutional network (2016, methodology): “To overcome this problem, we use a stack of unsupervised CAE with locally connected nodes and shared convolutional weights to extract local features from 3D images with possibly long voxel-wise signal vectors [19, 23, 24].”