From neural PCA to deep unsupervised learning

By Harri Valpola. 2014

195 citations recorded by the source corpus.

Abstract A network supporting deep unsupervised learning is presented. The network is an autoencoder with lateral shortcut connections from the encoder to the decoder at each level of the hierarchy. The lateral shortcut connections allow the higher levels of the hierarchy to focus on abstract invariant features. Whereas autoencoders are analogous to latent variable models with a single layer of stochastic variables, the proposed network is analogous to hierarchical latent variable models. Learning combines denoising autoencoder and denoising sources separation frameworks. Each layer of the network contributes to the cost function a term which measures the distance of the representations produced by the encoder and the decoder. Since training signals originate from all levels of the network, all layers can learn efficiently even in deep networks. The speedup offered by cost terms from higher levels of the hierarchy and the ability to learn invariant features are demonstrated in experiments.

Subfields: Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Signal Processing

Citations in the corpus

Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “The initial experimental evaluations of ladder networks [49] are limited to some standard tasks, such as handwritten digits classification over the Modified National Institute of Standards and Technology (MNIST) datasets [51] or image recognition tasks on the datasets of the Canadian Institute for A”
Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA (2016, methodology): “Some of the most popular methods are multi-layer belief nets and Restricted Boltzmann Machines [13] as well as autoencoders [14, 31, 21], which form the basis for the ladder networks [30].”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, methodology): “For the static face classification task, we compare the PredNet to a standard autoencoder and a Ladder Network [39, 50].”
Toward an Integration of Deep Learning and Neuroscience (2016, positive): “Ladder autoencoders make use of lateral connections and local noise injection to introduce an unsupervised cost function, based on internal reconstructions, that can be readily combined with supervised cost functions deﬁned on the networks top layer outputs (Valpola, 2015).”
Semi Supervised Semantic Segmentation Using Generative Adversarial Network (2017, methodology): “Unfortunately, unsupervised learning methods have not been very successful for semantic segmentation, because they lack the notion of classes and merely try to identify consistent regions and/or region boundaries [28].”
Recurrent Ladder Networks (2017, background): “Recurrent Ladder (RLadder) is a recurrent neural network whose units resemble the structure of the original Ladder networks [24, 31] (see Fig.”
Stacked Generative Adversarial Networks (2016, methodology): “Also, some auto-encoder architectures employ layer-wise reconstruction loss [60, 48, 67, 66].”
Improving Semi-Supervised Learning with Auxiliary Deep Generative Models (2015, background): “Finally Rasmus et al. (2015) introduced a generalization of the ladder network (Valpola, 2014) that has the ability to learn a latent classiﬁcation variable.”
An Overview of Deep Semi-Supervised Learning (2020, methodology): “[151] with an additional encoder and decoder for SSL.”
Ladder Variational Autoencoders (2016, methodology): “This parameterization allows interactions between the bottom-up and top-down signals resembling the recently proposed Ladder Network [21, 16], and we therefore denote it Ladder-VAE (LVAE).”
A Semisupervised Approach for Language Identification based on Ladder Networks (2016, methodology): “Our approach is based on the recently proposed Ladder Network training procedure that has proven to be very successful, especially in cases where the training dataset is composed of both labeled and unlabeled data [9] [10].”
Transductive Semi-Supervised Deep Learning Using Min-Max Features (2018, methodology): “Rasmus et al. [47] proposed a SSL method with ladder networks [48].”
Neural Networks and Deep Learning (2018, unknown): “Another form of semi-supervised learning can be performed with ladder networks [388, 502], in which skip-connections are used in conjunction with an autoencoder-like architecture.”
How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks (2016, background): “(11) contains the reconstruction term p θ ( x z ) and the variational regularization term.”