Semi-supervised Learning with Ladder Networks

By Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, Tapani Raiko. 2015

1,426 citations recorded by the source corpus.

We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on top of the Ladder network proposed by Valpola [1] which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification in addition to permutation-invariant MNIST classification with all labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, background): “Another intriguing line of work consists of the ladder network [14], which has achieved spectacular results on a semi-supervised variant of the MNIST dataset.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “Apart from self-training, another important line of work in semi-supervised learning [9, 85] is based on consistency training [6, 4, 53, 36, 70, 45, 41, 51, 10, 12, 49, 2, 38, 72, 74, 5, 81].”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “isms which can broadly be grouped into two categories: generative models that learn compact repre1 arXiv:1905.09272v1 [cs.CV] 22 May 2019 sentations that are capable of predicting pixel-level details [34,54,12], and self-supervised methods that formulate alternative prediction tasks in which missing”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “Many of the initial results on semi-supervised learning with deep neural networks were based on generative models such as denoising autoencoders [31], variational autoencoders [15] and generative adversarial networks [27, 33].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In the semi-supervised setting [55, 56], an unsupervised loss is combined with a classification loss over a handful of labels to ground the training [19, 20, 57, 58, 59, 60, 61, 62].”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “The range of applications that come with generative models are vast, where audio synthesis [55] and semi-supervised classification [38, 31, 44] are examples hereof.”
Improved Techniques for Training GANs (2016, methodology): “DGN[21] 333 ± 14 VirtualAdversarial[22] 212 CatGAN[14] 191 ± 10 SkipDeepGenerativeModel[23] 132 ± 7 Laddernetwork[24] 106 ± 37 AuxiliaryDeepGenerativeModel[23] 96 When presented with 50% real and 50% fake data generated by our best CIFAR-10 model, MTurk users correctly categorized 78 .”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “The deﬁnition of other hyperparameters are found in section 2. that are similar to FixMatch and/or are state-of-the-art: Π - Model [36], Mean Teacher [43], Pseudo-Label [22], Mix-Match [3], UDA [45], and ReMixMatch [2].”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, background): “In practice, semi-supervised learning methods with many hyperparameters can be problematic because cross-validation is difficult with small validation sets [35, 39, 35].”
Revisiting Self-Training for Neural Sequence Generation (2019, neutral): “These approaches have been very successful on classiﬁcation tasks (Rasmus et al., 2015; Miyato et al., 2017; Laine & Aila, 2017; Miyato et al., 2018; Xie et al., 2019) given that a reasonable amount of predictions of unlabeled data (at least the ones with high conﬁdence) are correct, but their…”
Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks (2015, methodology): “…5.21 2.67 - VAE+SVM (Kingma et al., 2014) 11.82 ( ± 0.25) 4.24 ( ± 0.07) - SS-VAE (Kingma et al., 2014) 3.33 ( ± 0.14) 2.4 ( ± 0.02) 0.96 Ladder Γ -model (Rasmus et al., 2015) compare the CatGAN algorithm with standard k-means clustering and RIM with neural networks as discriminative models,…”
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (2015, background): “…train auto-encoders (convolutionally, stacked (Vincent et al., 2010), separating the what and where components of the code (Zhao et al., 2015), ladder structures (Rasmus et al., 2015)) that encode an image into a compact code, and decode the code to reconstruct the image as accurately as possible.”
Adversarial Autoencoders (2015, methodology): “On the MNIST dataset with 100 and 1000 labels, the performance of AAEs is signiﬁcantly better than VAEs, on par with VAT [Miyato et al., 2015] and CatGAN [Springenberg, 2015], but is outperformed by the Ladder networks [Rasmus et al., 2015] and the ADGM [Maaløe et al., 2016].”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “A batch of novel models have been recently proposed for semi-supervised learning based on representation learning techniques, such as generative models (Kingma et al., 2014), ladder networks (Rasmus et al., 2015) and graph embeddings (Yang et al., 2016a).”
Densely Connected Convolutional Networks (2016, background): “The augmentation of networks with pathways that minimize reconstruction losses was also shown to improve image classiﬁcation models [42].”
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, methodology): “SWWAE [43] 23.56 Skip Generative Model [24] 16.30 Ladder networks, Γ model [30] 20.40 CatGAN [33] 19.58 GAN with feature matching [32] 8.11 18.63 model [21] 5 Table 5 shows the performance of VAT and contemporary methods implemented with moderate image data augmentation (translation and horizontal…”
Energy-based Generative Adversarial Network (2016, methodology): “Speciﬁcally, we utilize an identical architecture as reported in both papers (Rasmus et al., 2015; Pezeshki et al., 2015); namely a fully-connected network of size 784-1000-500-250-250-250 , with batch normalization and ReLU following each linear layer.”
Autoencoding beyond pixels using a learned similarity metric (2015, background): “Unfortunately, we have not been able to reach results competitive with the state-of-the-art (Rasmus et al., 2015; Zhao et al., 2015).”
Auxiliary Deep Generative Models (2016, background): “Lately the Ladder network (Rasmus et al., 2015; Valpola, 2014) and virtual adversarial training (VAT) (Miyato et al., 2015) have improved the performance further with end-to-end training.”